gem-hunter
Language: Go
pqstream turns your postgres database into an event stream
pqstream is a program that streams changes out of a postgres database with the intent of populating other systems and enabling stream processing of data sets.
$ go get -u github.com/tmc/pqstream/cmd/{pqs,pqsd}create an example database:
$ createdb dbname
# echo "create table notes (id serial, created_at timestamp, note text)" | psql dbnameconnect the agent:
$ pqsd -connect postgresql://user:pass@host/dbname
connect the cli:
$ pqs
at this point you will see streams of database operations rendered to stdout:
(in a psql shell):
dbname=# insert into notes values (default, default, 'here is a sample note'); INSERT 0 1 dbname=# insert into notes values (default, default, 'here is a sample note'); INSERT 0 1 dbname=# update notes set note = 'here is an updated note' where id=1; UPDATE 1 dbname=# delete from notes where id = 1; DELETE 1 dbname=#
our client should now show our operations:
$ pqs
{"schema":"public","table":"notes","op":"INSERT","id":"1","payload":{"created_at":null,"id":1,"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"INSERT","id":"2","payload":{"created_at":null,"id":2,"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"UPDATE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"},"changes":{"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"DELETE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"}}If there's a need to prevent sensitive fields (i.e. PII) from being exported the redactions flag can be used with pqsd:
$ pqsd -connect postgresql://user:pass@host/dbname -redactions='{"public":{"users":["first_name","last_name","email"]}}'The redactions is encoded in JSON and conforms to the following layout:
'{"schema":{"table":["field1","field2"]}}'`