bitcoin-s/docs/secp256k1/add-to-jni.md

id	title
jni-modify	Adding to Secp256k1 JNI

Bitcoin-S uses a Java Native Interface (JNI) to execute functions in secp256k1 from java/scala. The native java bindings used to be a part of the secp256k1 library that was maintained by bitcoin-core, but it was removed in October 2019. We maintain a fork of secp256k1 which forks off of bitcoin-core's master but re-introduces the jni. This is also the easiest way to add functionality from new projects such as Schnorr signatures and ECDSA adaptor signatures by rebasing the bitcoin-s branch with the JNI on top of these experimental branches. That said, it is quite tricky to hook up new functionality in secp256k1 into bitcoin-s and specifically NativeSecp256k1.java. The following is a description of this process.

• Adding a new function to NativeSecp256k1.java
  • Adding to src/java/org_bitcoin_NativeSecp256k1.c
  • Adding to src/java/org_bitcoin_NativeSecp256k1.h
  • Adding to src/java/org/bitcoin/NativeSecp256k1.java
  • Adding to src/java/org/bitcoin/NativeSecp256k1Test.java
  • Adding to Bitcoin-S
  • Further Work to Enable Typed Invocations and Nice Tests

Adding a new function to NativeSecp256k1.java

Adding to src/java/org_bitcoin_NativeSecp256k1.c

1. Add an #include import at the top (if applicable)

   If your secp256k1 functions are not already included, you will need to #include the header file (should be in the secp256k1/include directory).

2. Function signature

   Your new function signature should begin with

   SECP256K1_API jint JNICALL Java_org_bitcoin_NativeSecp256k1_
   

   followed by the secp256k1 function name where _ are replaced with _1 (it's a weird jni thing). Finally, you add a parameter list that begins with

   (JNIEnv* env, jclass classObject, jobject byteBufferObject, jlong ctx_l
   

   and ends with any jints in the case that any of the secp256k1 function inputs have variable length (such as public keys which can be either 33 or 65 bytes, or ECDSA signatures), and lastly any jbooleans in case there is some flag like compressed passed in.

   As an example that includes everything, if you are making a call to secp256k1_pubkey_tweak_add which takes in public keys that could be 33 or 65 bytes and outputs a public key that will either be compressed or decompressed based on an input flag, then the function signature would be

   SECP256K1_API jobjectArray JNICALL Java_org_bitcoin_NativeSecp256k1_secp256k1_1pubkey_1tweak_1add
     (JNIEnv* env, jclass classObject, jobject byteBufferObject, jlong ctx_l, jint publen, jboolean compressed)
   

3. Reading unsigned char* inputs

   It is now time to create pointers for each of the secp256k1 function inputs that where passed in via the byteBufferObject. We must first read in the Secp256k1Context with the line

   secp256k1_context *ctx = (secp256k1_context*)(uintptr_t)ctx_l;
   

   and we can then initialize the first pointer to be the beginning of the byteBufferObject with the line

   unsigned char* firstArg = (*env)->GetDirectBufferAddress(env, byteBufferObject);
   

   and subsequent arguments' pointers where the previous argument's length is known (say 32 bytes for example) can be instantiated using

   unsigned char* prevArg = ...
   unsigned char* nextArg = (unsigned char*) (prevArg + 32);
   

   and in the case that a previous argument has variable length, then a jint has been provided as an input and can be used instead, such as in the example

   unsigned char* prevArg = ...
   unsigned char* nextArg = (unsigned char*) (prevArg + publen);
   

   where publen is a jint passed to this C function.

   As an example that includes everything, consider the function secp256k1_ecdsa_verify which takes as input a 32 byte message, a variable length signature and a public key (of length 33 or 65 bytes). Our function will begin with

   {
     secp256k1_context *ctx = (secp256k1_context*)(uintptr_t)ctx_l;
   
     unsigned char* data = (unsigned char*) (*env)->GetDirectBufferAddress(env, byteBufferObject);
     const unsigned char* sigdata = (unsigned char*) (data + 32);
     const unsigned char* pubdata = (unsigned char*) (sigdata + siglen);
   

   where siglen is a jint passed into the C function.

4. Initialize variables

   Next we must declare all variables. We put all decelerations here as it is required by the C framework used by libsecp256k1 that definitions and assignments/function calls cannot be interleaved.

   Specifically you will need to declare any secp256k1 specific structs here as well as all outputs (such as jobjectArrays and jByteArrays). Generally speaking this will include all inputs which are not raw data (public keys, signatures, etc). Lastly, you will also have an int ret which will store 0 if an error occurred and 1 otherwise.

   As an example that includes everything, consider again the function secp256k1_pubkey_tweak_add has the following declarations

   jobjectArray retArray;
   jbyteArray pubArray, intsByteArray;
   unsigned char intsarray[2];
   unsigned char outputSer[65];
   size_t outputLen = 65;
   secp256k1_pubkey pubkey;
   int ret;
   

   Where retArray is eventually going to be the data returned, which will contain the jbyteArrays pubArray and intsByteArray, which will contain outputSer and intsarray respectively. Lastly pubkey will store a deserialized secp256k1_pubkey corresponding to the input unsigned char* public key.

5. Parse inputs when applicable

   In the case where there are unsigned char* inputs which need to be deserialized into secp256k1 structs, this is done now. As an example, secp256k1_pubkey_tweak_add takes a public key as input:

   unsigned char* pkey = (*env)->GetDirectBufferAddress(env, byteBufferObject);
   

   where a jint publen is passed in as a function parameter. This function already has a declaration for secp256k1_pubkey pubkey;. The first call made after the above declarations is

   ret = secp256k1_ec_pubkey_parse(ctx, &pubkey, pkey, publen)
   

   and if further parsing is necessary, it is put inside of if (ret) { ret = [further parsing here] }.

6. Make calls to secp256k1 functions to instantiate outputs

   It is finally time to actually call the secp256k1 function we are binding to the jni! This is done by simply calling ret = [call to secp function here]; or if (ret) { ret = [secp function call] }; if there were any inputs that needed to be parsed. Note that some secp256k1 functions return outputs by populating variables you should have declared and for which pointers are passed as inputs, while other functions will mutate their inputs rather than returning outputs.

7. Serialize variable length outputs if applicable

   When dealing with variable length outputs such as signatures, you will likely need to serialize these outputs. This is done by having already instantiated such a variable as

   unsigned char outputSer[72];
   size_t outputLen = 72;
   

   where in this case 72 is an upper bound on signature length. With these variables existing (as well as a secp256k1_ecdsa_signature sig which has been populated), we call a secp256k1 serialization function to populate outputSer and outputLen from sig

   if(ret) {
       int ret2 = secp256k1_ecdsa_signature_serialize_der(ctx, outputSer, &outputLen, &sig);
       (void)ret2;
     }
   

   As you can see, in this case we do not which to alter the value returned in ret if serialization fails. If we did then ret2 would not be introduced and we would instead do ret = [serialize].

8. Populate return array when applicable

   We now begin translating our serialized results back into Java entities. If you are returning any ints containing meta-data (usually ret is included here, as are the variable lengths of outputs when applicable), you will want an unsigned char intsarray[n] to be already declared where n is the number of pieces of meta-data. For example, in secp256k1_ecdsa_sign, we wish to return whether there were any errors (stored in int ret) and the output signature's length, size_t outputLen. Hence we have an unsigned char intsarray[2] and we populate it as follows

   intsarray[0] = outputLen;
   intsarray[1] = ret;
   

   Next we populate the jobjectArray we wish to return, this will always begin with a call

   retArray = (*env)->NewObjectArray(env, 2,
     (*env)->FindClass(env, "[B"),
     (*env)->NewByteArray(env, 1));
   

   to instantiate an empty return array. Next we instantiate our jbyteArrays with calls to

   myByteArray = (*env)->NewByteArray(env, len);
   

   where myByteArray is replaced with a real name (such as intsByteArray) and len is replaced with the length of this array (either a constant or a populated size_t variable). Next we populate this array with our data by calling

   (*env)->SetByteArrayRegion(env, myByteArray, 0, len, (jbyte*)myData);
   

   where myData is a C array of unsigned char (of length len). Lastly, we place myByteArray into its place in retArray with

   (*env)->SetObjectArrayElement(env, retArray, index, myByteArray);
   

   where index is a constant (0, 1, 2, etc.) for the index of myByteArray within retArray. Note that you should follow our conventions and have index 0 contain the actual data to be returned and index 1 (and onward) contain any meta-data.

   Please note that if you wish not to return such meta-data (such as if you wish to return only a boolean), then none of the code in this subsection is required

9. void classObject

   Once we are ready to return, we first void the input classObject by making the call

   (void)classObject;
   

10. Return array when applicable, ret when applicable

    Lastly, we return retArray in the case where we wish to return a byte[], or ret in the case that we wish to return a boolean.

Adding to src/java/org_bitcoin_NativeSecp256k1.h

Once your function is defined in src/java/org_bitcoin_NativeSecp256k1.c, you must define them in the corresponding header files by simply copying the function signature but without parameter names. For example, if secp256k1_pubkey_tweak_add has the function signature

SECP256K1_API jobjectArray JNICALL Java_org_bitcoin_NativeSecp256k1_secp256k1_1pubkey_1tweak_1add
  (JNIEnv* env, jclass classObject, jobject byteBufferObject, jlong ctx_l, jint publen, jboolean compressed)

then in the header file we include

/*
 * Class:     org_bitcoin_NativeSecp256k1
 * Method:    secp256k1_pubkey_tweak_add
 * Signature: (Ljava/nio/ByteBuffer;JI)[[B
 */
SECP256K1_API jobjectArray JNICALL Java_org_bitcoin_NativeSecp256k1_secp256k1_1pubkey_1tweak_1add
  (JNIEnv *, jclass, jobject, jlong, jint, jboolean);

Adding to src/java/org/bitcoin/NativeSecp256k1.java

We are now done writing C code! We have completed an interface in C for the JNI to hook up to. However, we must now write the corresponding Java code which hides the Java to C (and back) conversions from other Java code. We accomplish this with a class of static methods called NativeSecp256k1.

1. Add private static native secp256k1 function

   We begin by adding a private static native method at the bottom of the file corresponding to our secp256k1 function. Notice that the syntax for native methods is similar to that of Java abstract interface methods where instead of providing an implementation we simply end with a semi-colon.

   For functions returning booleans, we have their native methods return int (will be 0 or 1). Otherwise, for functions returning byte[]s, we have their native methods return byte[][] (two dimensional array to allow for meta-data).

2. Method signature

   Next we add a method to the NativeSecp256k1 class

   public static byte[] myFunc(byte[] input1, byte[] input2, boolean input3) throws AssertFailException
   

   where boolean could also be the return type instead of byte[].

3. checkArguments

   We begin implementing this function by checking the input argument lengths using the checkArument function

   checkArgument(input1.length == 32 && (input2.length == 33 || input2.length == 65));
   

4. Initialize ByteBuffer

   We now initialize the ByteBuffer which we will be passing through the JNI as an input. This is done with a call to

   ByteBuffer byteBuff = nativeECDSABuffer.get();
   

   followed by allocation when necessary as follows

   if (byteBuff == null || byteBuff.capacity() < input1.length + input2.length) {
       byteBuff = ByteBuffer.allocateDirect(input1.length + input2.length);
       byteBuff.order(ByteOrder.nativeOrder());
       nativeECDSABuffer.set(byteBuff);
   }
   

   where input1.length + input2.length is replaced by whatever the total ByteBuffer length needed.

5. Fill ByteBuffer

   We now populate the ByteBuffer as follows

   byteBuff.rewind();
   byteBuff.put(input1);
   byteBuff.put(input2);
   

   where generally, you will rewind() and then put() all inputs (in order).

6. Make native call

   It is now time to make a call to our native C function.

   In the case where we are returning a byte[], this is done by first declaring a byte[][] to store the output and then locking the read lock, r. Then we call the native function within a try clause which releases the lock in the finally clause.

   byte[][] retByteArray;
   r.lock();
   try {
       retByteArray = secp256k1_my_call(byteBuff, Secp256k1Context.getContext(), input3);
   } finally {
       r.unlock();
   }
   

   In the case where we are returning a boolean, simply make the call in the try and compare the output to 1 like so

   r.lock();
   try {
       return secp256k1_my_bool_call(byteBuff, Secp256k1Context.getContext()) == 1;
   } finally {
       r.unlock();
   }
   

   If this is the case, you are now done and can ignore the following steps.

7. Parse outputs

   retByteArray should now be populated and we want to read its two parts (data and meta-data). Getting the data should be as easy as

   byte[] resultArr = retByteArr[0];
   

   while for each piece of meta-data, you can read the corresponding int as follows

   int metaVal = new BigInteger(new byte[] { retByteArray[1][index] }).intValue();
   

   where index is replaced with the index in the meta-data array.

8. Validate outputs

   In the case where we now have meta-data, we validate it with calls to assertEquals.

9. Return output

   Finally, we return resultArr.

Adding to src/java/org/bitcoin/NativeSecp256k1Test.java

I normally first build the C binaries and add to Bitcoin-S before coming back to this section because I use sbt core/console to generate values and make calls below, but this is not a requirement.

1. Generate values and make calls to org.bitcoin.NativeSecp256k1 to generate inputs and their expected outputs

2. Create regression unit tests with these values in NativeSecp256k1Test

   Note that you can use DatatypeConverter.parseHexBinary to convert String hex to a byte[], and you can use DatatypeConverter.printHexBinary to convert a byte[] to its String hex. Lastly you will make assertions with calls to assertEquals.

3. Add test to main

Adding to Bitcoin-S

1. Translate NativeSecp256k1 and NativeSecp256k1Test to jni project

   By translate I mean to say that you must copy the functions from those files to the corresponding files in the bitcoin-s/secp256k1jni project. For tests this will require changing calls to DatatypeConverter to either toByteArray or toHex as well as changing the method to make it non-static as well as adding the @Test annotation above the method (rather than adding to a main method).

2. Configure and build secp256k1

   You will need to go to the bitcoin-s/secp256k1 directory in a terminal and running the following where you may need to add to the ./configure command if you are introducing a new module.

   For Linux or OSx (64-bit)

   You will have to make sure JAVA_HOME is set, and build tools are installed, for Linux this requires:

   echo JAVA_HOME
   sudo apt install build-essential autotools-dev libtool automake
   

   and for Mac this requires:

   brew install automake libtool
   

   You should then be able to build libsecp256k1 with the following:

   ./autogen.sh
   ./configure --enable-jni --enable-experimental --enable-module-ecdh
   make
   make check
   make check-java
   

   For Windows (64-bit)

   Windows bindings are cross-built on Linux. You need to install the mingw toolchain and have JAVA_HOME point to a Windows JDK:

   sudo apt install g++-mingw-w64-x86-64
   sudo update-alternatives --config x86_64-w64-mingw32-g++
   

   You should then be able to build libsecp256k1 with the following:

   echo "LDFLAGS = -no-undefined" >> Makefile.am
   ./configure --host=x86_64-w64-mingw32 --enable-experimental --enable-module_ecdh --enable-jni && make clean && make CFLAGS="-std=c99"
   

3. Copy binaries into bitcoin-s natives for your system

   You have now built the C binaries for your JNI bindings for your operating system and you should now find your operating system's directory in bitcoin-s/secp256k1jni/natives and replace its contents with the contents of secp256k1/.libs (which contains the compiled binaries).

4. Run secp256k1jni tests

   If you have not yet implemented tests, you should now be able to go back and do so as calls to NativeSecp256k1 should now succeed.

   Once you have tests implemented, and assuming you've copied them correctly to the bitcoin-s/secp256k1jni project, you should be able to run them using

   sbt secp256k1jni/test
   

Further Work to Enable Typed Invocations and Nice Tests

1. Add new NetworkElements where applicable

   In the case where you are dealing in new kinds of data that are not yet defined in Bitcoin-S, you should add these as case classes extending the NetworkElement trait, and give them companion objects extending Factory for easy serialization and deserialization.

   This step is not necessary if you are only dealing in raw data, ECPrivateKeys, ECPublicKeys, etc.

2. Add new typed functions to relevant data types where applicable

   In the case where your new function should be a static method, find a good object (or introduce one) and give it a def which takes in typed arguments and outputs typed arguments (using ByteVector in all places dealing with raw data rather than using Array[Byte]). You will then implement these methods using calls to NativeSecp256k1 methods and getting the inputs into Array[Byte] form by getting their ByteVectors (usually through a call to _.bytes) and then calling _.toArray.

   You will then need to take the data returned and deserialize it.

   In the case where your new function belongs naturally as an action performed by some existing or newly introduced type, you can implement your new function as a call made by that class as described for the previous case but where the class will pass a serialized version of itself into the NativeSecp256k1 call.

   It is often acceptable to implement the call in an object and then also add the call (via a call to the object, passing this) to the interface of relevant types.

3. Implement Bouncy Castle fallback in BouncyCastleUtil.scala if you can.

4. Add unit and property-based tests.

5. If you implemented Bouncy Castle fallback, add tests to BouncyCastleSecp256k1Test to compare implementations