Persistant FHEVM native blockchain

Zama fhEVM
1

Hi,
I wonder if it’s possible to have a persistant fhevm native blockchain, based on the docker compose at fhevm-devops release 0.5.x ?
I’ve played around with the docker-compose file, fhevm-devops/docker-compose/docker-compose-full.yml especially with the fhevm validator container :

fhevm-validator:
    environment:
      - TFHE_EXECUTOR_CONTRACT_ADDRESS=0x05fD9B5EFE0a996095f42Ed7e77c390810CF660c
    image: ghcr.io/zama-ai/ethermint-dev-node:v0.5.1-2
    ports:
      - "26656-26657:26656-26657"
      - "8545-8546:8545-8546"
    volumes:
      - $PWD/network-fhe-keys:/network-fhe-keys:Z
      - ../setup.sh:/config/setup.sh:Z
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: "curl -s -H \"Connection: Upgrade\" -H \"Upgrade: websocket\"  http://localhost:8546"
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 20s

I’m no expert in docker, I’ve tried to create a persistant volume for the fhevm-validator image (mirroring /root/.ethermintd in the container, that’s where the validator store blockchain data) but it fails on second startup (when I try to restart the fhevm-native chain where it stopped).

- $PWD/blockchain_data:/root/.ethermintd:Z # Added volume mapping for blockchain data

It looks like the docker image of the validator is setup to execute genesis at startup.
I’ve also tried to play with setup.sh and even removing it from the docker image, by commenting the line

 - ../setup.sh:/config/setup.sh:Z

In the docker compose, the fhevm-validator still fails at startup.

I’m maybe missing something, but I feel like this repo was not meant to be persistant, it’s better for dev purpose to start on a fresh setup everytime you restart the docker containers.
However I wanted to know if it’s possible.

PS : I know the fhevm-devops repo 0.5.x is “deprecated” and that it will be probably easier to achieve persistance using a classic EVM L1 and the coprocessor docker, but I’m still interested in FHEVM-native blockchain.

Thanks !

2

Hello!
This image is for dev purpose but you can try with ethermint-node v0.5.1-2 instead.

3

Thank you very much, looks like this image is fine for what I want.

I was able to restart block mining at a non 0 height.

The process is a bit sketchy at the moment :
I’m starting a clean environment with the dev node image, it starts at genesis.
When this original chain, started with the dev node image, is stopped, I switch with the image you provided me before restarting it.
The new validator then starts well and does its job by resuming at a non 0 block height.

For the moment I’m not able to use the faucet on the non-dev node image (after restarting the chain).

INFO: global keys aren't initialized automatically (FHEVM_GO_KEYS_DIR env variable not set)

I’ll try to pass some environment variables to the container or make it execute a script before starting the validator.

I’m looking into it and will come back here if I need further help.
Thanks again for sharing this image !

4

Hello again,

I have some questions to ask. First let me give you some context on how my tests are executed :

  • Step 1 : Start a new FHEVM blockchain (no blockchain historical data) with the dev node image, in order to execute genesis using the make run-full command.
  • Step 2 : Once the new blockchain is started, execute the make prepare-e2e-test in order to publish all the needed contracts on the chain.
  • Step 3 : Deploy a classic EncryptedERC20 and storing its address for later use.
  • Step 4 : Stopping the created chain using the make stop-full command.
  • Step 5 : Rebooting the chain using make run-full, this time an if statement in the makefile determines there is historical blockchain data and changes the dev node image to the image you provided me (ethermint-node v0.5.1-2).
  • Step 6 : Creating an instance of the contract previously deployed using the same signer and the previously saved contract address.
  • Step 7 : Mint tokens.

I am failing at Step 7. I can, however, read from the contract (calling balanceOf - it returns 0 of course) after rebooting the chain as described. I assume the error is created when the validator tries to execute code using TFHE solidity library.

The error I get is the following (when the mint function is called by my hardhat test unit) :

Error: socket hang up
      at Socket.socketOnEnd (node:_http_client:536:25)
      at Socket.emit (node:events:525:35)
      at Socket.emit (node:domain:489:12)
      at endReadableNT (node:internal/streams/readable:1696:12)
      at processTicksAndRejections (node:internal/process/task_queues:90:21)

Crash logs from the validator container (resized because it’s way too long) :

thread 'ethermintd' panicked at tfhe/src/c_api/high_level_api/keys.rs:62:54:
called `Result::unwrap()` on an `Err` value: "pointer is null, got: 0x0"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
ethermintd: tfhe_wrappers.c:53: checked_set_server_key: Assertion `r == 0' failed.
SIGABRT: abort
PC=0x7fc8e755be2c m=8 sigcode=18446744073709551610
signal arrived during cgo execution
	
goroutine 692 gp=0xc003112fc0 m=8 mp=0xc000500008 [syscall]:
runtime.cgocall(0x2483fc0, 0xc0051bb380)
	runtime/cgocall.go:157 +0x4b fp=0xc0051bb358 sp=0xc0051bb320 pc=0x4b058b
github.com/zama-ai/fhevm-go/fhevm/tfhe._Cfunc_trivial_encrypt_fhe_uint64(0x0, 0x0) 
...

My docker-compose-full file looks like this :

name: zama-dev

services:

  gateway-store:
    image: ghcr.io/zama-ai/kms-blockchain-gateway-dev:v0.8.2-rc4
    command:
      - "kv_store"
    ports:
      - "8088:8088"

  kms-validator:
    image: ghcr.io/zama-ai/kms-blockchain-asc-dev:v0.8.2-rc4
    ports:
      - "36656:26656"
      - "36657:26657"
      - "1317:1317"
      - "9090:9090"
    entrypoint: [ "/app/bootstrap.sh" ]
    healthcheck:
      test: "wget -Sq --spider http://localhost:26657/status"
      interval: 1s
      timeout: 1s
      retries: 5
      start_period: 10s

  connector:
    image: ghcr.io/zama-ai/kms-blockchain-connector-dev:v0.8.2-rc4
    command:
      - "kms-blockchain-connector"
    environment:
      - ASC_CONN__BLOCKCHAIN__ADDRESSES=http://kms-validator:9090
      - ASC_CONN__CORE__ADDRESSES=http://kms-core:50051
      - ASC_CONN__STORE__URL=http://gateway-store:8088
      - ASC_CONN__CORE__TIMEOUT_CONFIG__DECRYPTION__INITIAL_WAIT_TIME=1
      - ASC_CONN__CORE__TIMEOUT_CONFIG__DECRYPTION__RETRY_INTERVAL=1
      - ASC_CONN__CORE__TIMEOUT_CONFIG__REENCRYPTION__INITIAL_WAIT_TIME=1
      - ASC_CONN__CORE__TIMEOUT_CONFIG__REENCRYPTION__RETRY_INTERVAL=1
    depends_on:
      kms-validator:
        condition: service_healthy
      kms-core:
        condition: service_healthy

  kms-core:
    image: ghcr.io/zama-ai/kms-service-dev:v0.8.2-rc4
    volumes:
      - $PWD/res/keys:/app/kms/core/service/keys:Z
    ports:
      - "50051:50051"
    healthcheck:
      test: "grpc-health-probe --addr=localhost:50051"
      interval: 1s
      timeout: 1s
      retries: 5
      start_period: 10s

  gateway:
    image: ghcr.io/zama-ai/kms-blockchain-gateway-dev:v0.8.2-rc4
    ports:
      - "7077:7077"
    command:
      - "gateway"
    volumes:
      - ../default.toml:/app/gateway/config/default.toml:Z
    environment:
      - GATEWAY__ETHEREUM__LISTENER_TYPE=FHEVM_V1_1
      - GATEWAY__ETHEREUM__WSS_URL=ws://fhevm-validator:8546
      - GATEWAY__ETHEREUM__HTTP_URL=http://fhevm-validator:8545
      - GATEWAY__ETHEREUM__FHE_LIB_ADDRESS=000000000000000000000000000000000000005d
      - GATEWAY__ETHEREUM__ORACLE_PREDEPLOY_ADDRESS=c8c9303Cd7F337fab769686B593B87DC3403E0ce
      - GATEWAY__KMS__ADDRESS=http://kms-validator:9090
      - GATEWAY__KMS__KEY_ID=408d8cbaa51dece7f782fe04ba0b1c1d017b1088
      - GATEWAY__STORAGE__URL=http://gateway-store:8088
      - ASC_CONN__BLOCKCHAIN__ADDRESSES=http://kms-validator:9090
      - GATEWAY__ETHEREUM__RELAYER_KEY=7ec931411ad75a7c201469a385d6f18a325d4923f9f213bd882bbea87e160b67
      - RUST_BACKTRACE=1
    depends_on:
      fhevm-validator:
        condition: service_healthy
      kms-validator:
        condition: service_healthy

  fhevm-validator:
    environment:
      - TFHE_EXECUTOR_CONTRACT_ADDRESS=0x05fD9B5EFE0a996095f42Ed7e77c390810CF660c
    image: ghcr.io/zama-ai/ethermint-dev-node:v0.5.1-2
    ports:
      - "26656-26657:26656-26657"
      - "8545-8546:8545-8546"
    volumes:
      - $PWD/network-fhe-keys:/network-fhe-keys:Z
      - ../setup.sh:/config/setup.sh:Z
      - $PWD/blockchain_data:/root/.ethermintd:Z # Added volume mapping for blockchain data
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: "curl -s -H \"Connection: Upgrade\" -H \"Upgrade: websocket\"  http://localhost:8546"
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 20s

And here is the override I’m applying to it when there is already historical block data :

name: zama-dev

services:
  fhevm-validator:
    image: ghcr.io/zama-ai/ethermint-node:v0.5.1-2

Am I missing something ? I feel like I’m close to my goal but I can’t figure what to do. Should I add some volumes to the other containers to keep track of the data they produce ?
There is definetly some data persistance as I can call a contract that has been deployed on a stoped instance of the FHEVM chain, problems are showing when some FHE logic is involved.