Launch the service on Google Cloud
This document describes steps required to host the Silent Network node on Google Cloud. Audience of this documents are DevOps, or software engineers, preferably with basic knowledge of Google Cloud and the Shell.
Install gcloud
We will use official Google Cloud CLI for most of the time. Install it from here
Create Project on Google Cloud
Follow official instructions here. You also need to connect billing account.
Enable Compute Engine API to allow start up the VM instances
Go to https://console.cloud.google.com/apis/library/compute.googleapis.com
Then click "Enable"
Enable secret manager api
https://console.cloud.google.com/apis/library/secretmanager.googleapis.com
Setup gcloud
Set the env variables that we will use over the course of this tutorial. The easiest way to do so, is to create a file with following content.
# Your project ID (not project number!)
export WORKLOAD_OPERATOR_PROJECT_ID=
# VM related parameters
export VM_ZONE=
export VM_NAME=operator
export VM_ADDRESS=
# Bucket where user's data shall be stored
export WORKLOAD_OPERATOR_OUTPUT_BUCKET_NAME=
export BUCKET_LOCATION=
# Do not change. Parameters for operator's container registry
export WORKLOAD_AUTHOR_PROJECT_ID=silent-network-426409
export REGISTRY_NAME=silent-network-registry
export CONTAINER_IMAGE=operator-container-image
export WORKLOAD_VERSION=v0.1.5
# Service account attached to the VM
export WORKLOAD_OPERATOR_SERVICE_ACCOUNT_NAME=operator-service-account
The empty variables are going to be set in following sections:
Project
Please fill up WORKLOAD_OPERATOR_PROJECT_ID
with the ID of the project you've created. Immediately after set valid project for gloud
CLI, so subsequent commands will operate on valid project.
Make sure
gcloud
is pointing to valid project by callinggcloud config set project $WORKLOAD_OPERATOR_PROJECT_ID
VM Zone
Please select a VM_ZONE
for the hosted VM in Europe. Go to Available regions and zones , filter location by Europe
and machine series by N2D
. Select one of available zones and set the VM_ZONE
variable.
VM IP address
Set the static IP for the VM instance.
Pick region for the IP address. Needs to be the same as the one selected for the VM. The
VM_ZONE
variable containszone
, whereas for IP theregion
is needed. To getregion
from azone
, simply discard last two characters.
export IP_REGION=${VM_ZONE%??}
Create static IP in this region:
gcloud compute addresses create operator-static-ip --region=$IP_REGION
Get the IP address that was assigned from previous step
gcloud compute addresses describe operator-static-ip --region=$IP_REGION --format='get(address)'
Save the IP address in
VM_ADDRESS
variable
Bucket
Set the WORKLOAD_OPERATOR_OUTPUT_BUCKET_NAME
. The name has to be globally unique, the simplest way to do so is to add random suffix:
echo workload-output-bucket-$(head /dev/urandom | tr -dc a-z0-9 | head -c10)
Select a location of a bucket from here. Choose one from Europe. Set the BUCKET_LOCATION
.
Save the configuration
With all variables set, save the file as env.operator.configuration
and in the console load the variables by:
> source env.operator.configuration
Create bucket
Create the bucket by calling:
gcloud storage buckets create gs://${WORKLOAD_OPERATOR_OUTPUT_BUCKET_NAME} --location=${BUCKET_LOCATION}
The read, write permissions to it are going to be set next.
Create Service Account
The Service Account will be attached to the VM, and used to access cloud resources. Create a service account by executing:
gcloud iam service-accounts create $WORKLOAD_OPERATOR_SERVICE_ACCOUNT_NAME
Next, set valid permissions for the account
# grant workload SA serviceAccountUser, so can create VM workload
gcloud iam service-accounts add-iam-policy-binding \
$WORKLOAD_OPERATOR_SERVICE_ACCOUNT_NAME@$WORKLOAD_OPERATOR_PROJECT_ID.iam.gserviceaccount.com \
--member=user:$(gcloud config get-value account) \
--role=roles/iam.serviceAccountUser
# grant workload SA the confidentialcomputing.workloadUser role, so it can generate an attestation token
gcloud projects add-iam-policy-binding $WORKLOAD_OPERATOR_PROJECT_ID \
--member=serviceAccount:$WORKLOAD_OPERATOR_SERVICE_ACCOUNT_NAME@$WORKLOAD_OPERATOR_PROJECT_ID.iam.gserviceaccount.com \
--role=roles/confidentialcomputing.workloadUser
# grant workload SA the logging.logWriter role, so we can check progress of the workload
gcloud projects add-iam-policy-binding $WORKLOAD_OPERATOR_PROJECT_ID \
--member=serviceAccount:$WORKLOAD_OPERATOR_SERVICE_ACCOUNT_NAME@$WORKLOAD_OPERATOR_PROJECT_ID.iam.gserviceaccount.com \
--role=roles/logging.logWriter
# grant workload SA storage.objectUser role, so can read delete and update from the bucket
gcloud storage buckets add-iam-policy-binding gs://${WORKLOAD_OPERATOR_OUTPUT_BUCKET_NAME} \
--member=serviceAccount:$WORKLOAD_OPERATOR_SERVICE_ACCOUNT_NAME@$WORKLOAD_OPERATOR_PROJECT_ID.iam.gserviceaccount.com \
--role=roles/storage.objectUser
# grant the secretmanager.secretAccessor and viewer role, so can read secrets from secret manager
gcloud projects add-iam-policy-binding $WORKLOAD_OPERATOR_PROJECT_ID \
--member=serviceAccount:$WORKLOAD_OPERATOR_SERVICE_ACCOUNT_NAME@$WORKLOAD_OPERATOR_PROJECT_ID.iam.gserviceaccount.com \
--role=roles/secretmanager.secretAccessor
gcloud projects add-iam-policy-binding $WORKLOAD_OPERATOR_PROJECT_ID \
--member=serviceAccount:$WORKLOAD_OPERATOR_SERVICE_ACCOUNT_NAME@$WORKLOAD_OPERATOR_PROJECT_ID.iam.gserviceaccount.com \
--role=roles/secretmanager.viewer
Share the Service account details with us
Provide us output of this command
echo $WORKLOAD_OPERATOR_SERVICE_ACCOUNT_NAME@$WORKLOAD_OPERATOR_PROJECT_ID.iam.gserviceaccount.com
Our team will grant to that Service Account access the container registry.
Setup the firewall
Enable http and https connection to the VM instance.
gcloud compute firewall-rules create allow-http \
--allow tcp:80 --target-tags http-server
gcloud compute firewall-rules create allow-https \
--allow tcp:443 --target-tags https-server
Put secrets to Secret Manager
Signing key for MPC protocol
Store the secret key in Google's Secret Manager, the Container will fetch it upon launch.
openssl rand 32 | gcloud secrets create SILENT_NETWORK_NODE_MSG_SIGN_KEY \
--replication-policy="automatic" \
--data-file=-
Launch the VM instance
Create an environment variables file (name it container-env-file
) that is going to be passed to the running container.
Set following variables:
KEYSHARE_STORAGE
should point to created bucket, withgs://
prefix. For example:gs://workload-output-bucket-xxxxxxxxxx
OPERATOR_ETH_ADDRESS
the ETH address derived from public ECDSA key generated during registration to the AVS
KEYSHARE_STORAGE=
OPERATOR_ETH_ADDRESS=
# Do not change
AGGREGATOR_PUBLIC_KEY=fbf777920d5853faa49d50216030ab9fa1ef263d588392d79fff078c5f0a371a
RUST_LOG=debug
LISTEN=0.0.0.0:80
ETH_RPC_URL=https://ethereum-holesky-rpc.publicnode.com
ETH_WS_URL=wss://ethereum-holesky-rpc.publicnode.com
# Addresses from official SilentShard AVS deployment (proxies) on Holesky
OPERATOR_STATE_RETRIEVER=0xD1aC03296bF5131BC592F005bE94319Bb02c778d
REGISTRY_COORDINATOR=0x76a49007EfB04d114D047095A49EdAF9A5dD334b
# https://holesky.eigenlayer.xyz/avs/0x7e06dbbe4fc83a1bccd20af3079e08f57c19cf34
DKLS_SERVICE_MANAGER=0x7e06DbBe4fC83a1bCcd20AF3079e08F57C19cf34
NODE_API_ADDRESS=0.0.0.0:80
Save the file as container-env-file
Start the VM. The container shall be fetched from the private registry, and then executed. The container-env-file
is used to set up variables.
gcloud compute instances create-with-container ${VM_NAME} \
--maintenance-policy=TERMINATE \
--zone=${VM_ZONE} \
--tags=http-server,https-server \
--container-image europe-docker.pkg.dev/$WORKLOAD_AUTHOR_PROJECT_ID/$REGISTRY_NAME/${CONTAINER_IMAGE}:${WORKLOAD_VERSION} \
--service-account=$WORKLOAD_OPERATOR_SERVICE_ACCOUNT_NAME@$WORKLOAD_OPERATOR_PROJECT_ID.iam.gserviceaccount.com \
--address=${VM_ADDRESS} \
--tags=http-server,https-server \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--machine-type=n2d-standard-16 \
--metadata google-logging-enabled=true \
--container-env=GC_PROJECT_ID=${WORKLOAD_OPERATOR_PROJECT_ID} \
--container-env-file ./container-env-file
Troubleshooting
Check if the VM and the container started up without any problems:
curl ${VM_ADDRESS}
The command should return text response:ok
Error: The zone 'XXX' does not have enough resources available to fulfill the request.
There might be unlikely event upon VM start to face the error ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS.
It means there are insufficient HW resources in the selected VM_ZONE
Please choose different zone as described in VM Zone chapter.
There is no way to move the IP, across regions, so if new VM_ZONE is in different region, release old IP by calling:
gcloud compute addresses delete operator-static-ip
And then create VM IP address in new region and try again.
Serial console
You can connect to serial port of the VM to read the logs of the Host OS before the container launches. It's helpful in case if container does not starts up.
# Enable serial port
gcloud compute instances add-metadata ${VM_NAME} \
--zone=${VM_ZONE} \
--metadata serial-port-enable=TRUE
# Connect to serial port to see the logging of the Host OS
gcloud compute connect-to-serial-port --zone=${VM_ZONE} ${VM_NAME}
If everything went well and the container started up, you should be able to see following logs:
[ 31.297213] konlet-startup[1705]: 2024/06/17 12:07:21 Starting a container with ID: 6c9f3fd7c8f89328d736ee7526d3f15648b94e5f6b897bc7de69bcbe693125ab
[ 31.395243] konlet-startup[1705]: 2024/06/17 12:07:21 Saving welcome script to profile.d
To exit the console, hit Enter
, and then type: ~.
Logs from the container
If the VM is running, the output from serial console suggests container started up correctly, but there is no response from curl,
SSH to the VM:
gcloud compute ssh --project=${WORKLOAD_OPERATOR_PROJECT_ID} --zone=${VM_ZONE} ${VM_NAME}
Find name of running container:
docker ps
It should be in a form: xxx-operator-xxxx
. Get the logs:
docker logs xxx-operator-xxxx
Share the VM instance information with us
Share following data with our team in order to be able to connect you to the MPC network
The IP address from
VM_ADDRESS
variable,The Operator ETH address
The
VM_ZONE
.Public part of the MPC signing key stored in Secret Manager. Read it from the logs of the running VM instance:
gcloud logging read \
"resource.type=gce_instance AND resource.labels.instance_id=$( gcloud compute instances describe --format "get(id)" ${VM_NAME} --zone=${VM_ZONE} )" \
--limit 100 \
--format json \
--freshness 1d | grep "Master VK from GC Secret Manager"
Example output:
"message": "2024-07-09T15:30:03.686380Z INFO el_party_svc::signer: Master VK from GC Secret Manager: \"21dba548b92ad65bfdc6328e98f5099c17702f75d1fd82c98cff81e5abc26e29\"\n",
Last updated