Implement Cobuild integration and Solana support

[XuJiandong]

For the detail of Solana support, please see XuJiandong#48

Comments for Cobuild reviewing:

• The main implementation is in cobuild.h.
• Otx(open transaction) implementation is included.
• The words "tested by" in cobuild.c means the branch is covered by corresponding test case.
• The lazy reader is implemented in cobuild.c. No matter how large the transaction is, the total memory consumption is fixed(~2K for global and ~4K for stack). We have test case for it.
• The precomputed table for secp256k1 is removed. It can save a lot of memory (~1M).
• molecule data structure is verified with lazy reader(See verfiy_WitnessLayout)
• There is message calculation flow in omnilock which doesn't exist in cobuild spec. One extra byte at start of seal is for further use.

[xxuejie]

I haven't finished checking the correctness part of the code.

But the more I look at this code, the more I think we failed to deliver a proper set of API in molecule-c2, and the accompanying Rust API at nervosnetwork/molecule#77.

Personally, I would picture 2 clearly defined, distinct eras of APIs where CKB smart contracts use:

• Initially, we had a bunch of APIs, like load_input, load_witness, load_cell, load_cell_by_field, etc. And you are expected to be experienced enough to mix the use of them to achieve validation
• With lazy loader, we should be able to reduce the effort to only a selected few APIs: load_script for loading current script, and load_transaction for accessing anything related to the enclosing transaction.

In the second era, I would expect a smart contract only needs to build a single data source containing the entire transaction, and in 99% of the case, CKB transaction will very small, meaning a single syscall can be used by the data source to load the whole transaction into a memory buffer owned by the data source. Later the smart contract only needs to deal with a single cursor built from the data source containing the full transaction. The smart contract can then slice the cursor into smaller cursors, like one cursor for an input cell, and another cursor for an output cell lock, and yet another cursor for a witness. A cursor representing a witness can be further reassembly into a cursor for a WitnessLayout structure, or a WitnessArgs structure for legacy layout.

To me, that would bring us a much cleaner and a much performant design.

However, in current omnilock, we are still mixing numerous APIs together, littered with different patterns. get_witness_layout and ckb_fetch_otx_start are 2 examples here. We are manually coding against many hidden jargons, making the code harder to read nor maintain. Combining with the fact that molecule-c2 does no verification on the data structure, I must say I wouldn't personally agree with the way this lock is organized now.

Yes there is a chance the lock script might still be correct mostly and secure, but to me this lock is moving into a uncertain direction, and I'm afraid it's gonna bite us one day.

So if I were doing this, I would go back to molecule-c2, if the API is not good enough for usage, we should alter molecule-c2 with better APIs. After that, omnilock should only need to construct a single cursor containing the full CKB transaction, and continue to validate from there.

[XuJiandong]

In the second era, I would expect a smart contract only needs to build a single data source containing the entire transaction

A single data source is not working for following data structure:

1. cell_deps
2. header
3. inputs

[xxuejie]

In the second era, I would expect a smart contract only needs to build a single data source containing the entire transaction

A single data source is not working for following data structure:

1. cell_deps
2. header
3. inputs

Correct, those syscalls are just like load_script, which is outside of a single transaction, and should be loaded via designated syscalls.

But that does not void the reasoning here: there are simply too many code pieces in the implementation now to load a part of the data which indeed belong to the enclosing transaction, causing enough confusions and chaos.

[xxuejie]

This can of course be a latter task, but I believe we can move cobuild.c / cobuild.h (ideally I would personally just keep a cobuild.h file containing both declaration and implementation for the ease of integration, but that a question of preference) to ckb-c-stdlib for reuse in other contracts.

[homura]

Hi there, it's Lumos. We're planning to integrate with the CoBuild protocol through Omnilock and would like to ask a few questions before proceeding:

• Is there any documentation or example that demonstrates how Omnilock supports the CoBuild protocol
• Is the PR stable and unlikely to undergo major changes that could affect the integration process
• Do you have any other suggestions for SDKs that would be useful in integrating with this PR

[XuJiandong]

Hi there, it's Lumos. We're planning to integrate with the CoBuild protocol through Omnilock and would like to ask a few questions before proceeding:

• Is there any documentation or example that demonstrates how Omnilock supports the CoBuild protocol
• Is the PR stable and unlikely to undergo major changes that could affect the integration process
• Do you have any other suggestions for SDKs that would be useful in integrating with this PR

issue 2: yes, it's stable.
issue 3: We have test vectors in this PR. It will be useful.

[EthanYuan]

Hello, CKB SDK Rust is also planning to support CoBuild , it would be very helpful to have a synchronized PR for updating 0042-omnilock.md.

[doitian]

Contract usually only needs WitnessLayout, I would recommend spliting the schema into two files:

• witness_layout.mol: Contains WitnessLayout and its dependencies
• building_packet.mol: This is for offchain, it imports witness_layout and includes the definition for BuildingPacket as well as other dependencies.

[homura]

I have a question regarding the script built by make all-via-docker

I remember that the printf instructions used to be ignored or erased by the compiler. However, the compiled binary from this PR still contains the printf instructions. I am not sure if this is the expected behavior.

I received the following output from the ckb-debugger, where the log No otx detected may have been generated from this line:

No otx detected
message(len=12): 0c0000000800000004000000
seal(len=86): 00550000001000000055000000550000004100000016c86c5dfed70f32cf0b62e31517b7b376de7453a4e1637d7d4300f374a6ef7061183fef1f116073e2b235496ca96a2c2eb8b9c03e76cb0b5b88b427bfe0953a01
ckb_generate_smh total hashed 200 bytes
smh(len=32): 55009e27578c58aaf1255a713d525770f0992a9c2e09604fba4221c44fe2dded
cobuild_activated = 1
Run result: 0
Total cycles consumed: 1368092(1.3M)
Transfer cycles: 30437(29.7K), running cycles: 1337655(1.3M)

---

another possibility is that the printf instruction is not erased, but that the debugger I used before may not set an output printer, whereas the current debugger has one

[XuJiandong]

I have a question regarding the script built by make all-via-docker

I remember that the printf instructions used to be ignored or erased by the compiler. However, the compiled binary from this PR still contains the printf instructions. I am not sure if this is the expected behavior.

I received the following output from the ckb-debugger, where the log No otx detected may have been generated from this line:

No otx detected
message(len=12): 0c0000000800000004000000
seal(len=86): 00550000001000000055000000550000004100000016c86c5dfed70f32cf0b62e31517b7b376de7453a4e1637d7d4300f374a6ef7061183fef1f116073e2b235496ca96a2c2eb8b9c03e76cb0b5b88b427bfe0953a01
ckb_generate_smh total hashed 200 bytes
smh(len=32): 55009e27578c58aaf1255a713d525770f0992a9c2e09604fba4221c44fe2dded
cobuild_activated = 1
Run result: 0
Total cycles consumed: 1368092(1.3M)
Transfer cycles: 30437(29.7K), running cycles: 1337655(1.3M)

another possibility is that the printf instruction is not erased, but that the debugger I used before may not set an output printer, whereas the current debugger has one

It is expected behavior. Will be removed when it's ready. It's very important for troubleshooting now.

[homura]

It is expected behavior. Will be removed when it's ready. It's very important for troubleshooting now.

I see, it does helpful for debugging

[homura]

I noticed that during the scanning process, the contract checks for the presence of a WitnessLayout in witnesses to determine if CoBuild mode is activated. However, the current implementation seems incomplete in scenarios where an NFT can be minted while unlocking a long-term Nervos DAO. Such transactions may look like this:

inputs:
  - input#0:
      lock: omnilock
      type: dao
  - input#1 # mint lock cell
      lock: nft-mint-lock # to check if a Nervos DAO has been locked for a particular period
outputs:
  - output#0:
      lock: omnilock
      type: dao-withdraw
  - output#1 # mint lock cell
  - output#2 # NFT cell
witnesses:
  - witness#0: WitnessArgs # dao deposit header witness
  - witness#1: WitnessLayout.SighashAll # with mint NFT action

[XuJiandong]

Pudge testnet deployment information:

parameter	value
code_hash	0x533cbfca1aba7b6286d07c23292f80b89f690786bc1cc07c96f0b6643849fc5d
hash_type	type
tx_hash	0xbf5be776ebc34bf681606a197116eaf5f4b23ada55cc5fc32f7abbda081d0918
index	0x0
dep_type	code