gem-hunter
Language: Python
Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes
Streams the following Ethereum entities to Pub/Sub or Postgres using ethereum-etl stream:
Streams blocks and transactions to Pub/Sub using bitcoin-etl stream. Supported chains:
gcloud container clusters create ethereum-etl-streaming \ --zone us-central1-a \ --num-nodes 1 \ --disk-size 10GB \ --machine-type custom-2-4096 \ --network default \ --subnetwork default \ --scopes pubsub,storage-rw,logging-write,monitoring-write,service-management,service-control,trace
kubectl credentials:gcloud container clusters get-credentials ethereum-etl-streaming \ --zone us-central1-a
create_pubsub_topics_ethereum.sh). Skip this step if you need to stream to Postgres.Create GCS bucket. Upload a text file with block number you want to start streaming from to
gs://<YOUR_BUCKET_HERE>/ethereum-etl/streaming/last_synced_block.txt.
Create "ethereum-etl-app" service account with roles:
Download the key. Create a Kubernetes secret:
kubectl create secret generic streaming-app-key --from-file=key.json=$HOME/Downloads/key.json -n ethbrew install helm helm init bash patch-tiller.sh
values dir and adjust all the files at least with your bucket and project ID.helm install --name btc --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/bitcoin/values.yaml helm install --name bch --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/bitcoin_cash/values.yaml helm install --name dash --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/dash/values.yaml helm install --name dogecoin --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/dogecoin/values.yaml helm install --name litecoin --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/litecoin/values.yaml helm install --name zcash --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/zcash/values.yaml helm install --name eth-blocks --namespace eth charts/blockchain-etl-streaming \ --values values/ethereum/values.yaml --values values/ethereum/block_data/values.yaml helm install --name eth-traces --namespace eth charts/blockchain-etl-streaming \ --values values/ethereum/values.yaml --values values/ethereum/trace_data/values.yaml
Ethereum block and trace data streaming are decoupled for higher reliability.
To stream to Postgres:
helm install --name eth-postgres --namespace eth charts/blockchain-etl-streaming \
--values values/ethereum/values-postgres.yamlRefer to https://github.com/blockchain-etl/ethereum-etl-postgres for table schema and initial data load.
describe command to troubleshoot, f.e.:kubectl describe pods -n btc kubectl describe node [NODE_NAME]
Refer to blockchain-etl-dataflow for connecting Pub/Sub to BigQuery.