gem-hunter
Language: TypeScript
The universal GraphQL API and CSPM tool for AWS, Azure, GCP, K8s, and tencent.
CloudGraph is the free open-source universal GraphQL API and Cloud Security Posture Management (CSPM) tool for AWS, Azure, GCP, and K8s. With CloudGraph you get:
Cloud Graph lets you Know your cloud in 5 minutes. Built and maintained with love by the team at ❤️ AutoCloud ❤️
🌐 Website
💰 Get paid to build CloudGraph providers
** usage does not imply endorsement
AWS, Azure, and GPC have done a wonderful job of building solutions that let engineers like us create systems to power our increasingly interconnected world. Over the last 15 years, products such as EC2, S3, RDS, and Lambda have fundamentally changed how we think about computing, storage, and databasing.
With the proliferation of Kubernetes and Serverless in the last 5 or so years, cloud services have become increasingly abstract on top of racks of physical servers. To end-users, everything on the cloud is just an API, so we don't necessarily need to know how Lambda Functions or EKS work under the hood to be able to use them for building applications. With a little documentation, API or console access, and a tutorial anyone can pretty much create anything they need.
These abstractions have led to massive improvements in the overall convenience and breadth of CSP service offerings. What was once a painstaking, time-consuming, and error-prone process of provisioning new servers, databases, or filesystems can now be done in seconds with just the click of a button or deployment of IAC. Since everything is just an API abstraction, when a CAP is ready to introduce a new "product" they simply need to expose a new API - yes, I'm of course simplifying slightly :)
Anyone familiar with the CSPs knows that service APIs are almost always split into modular namespaces that contain dozens, if not hundreds, of separate API methods for single resources. For example, the AWS EC2 service contains over 500 different API methods, with new ones added occasionally. Any company building substantial systems on a CSP is likely using many, many different services.
While a masterpiece of datacenter architecture, this choice of hundreds of services and configuration options put the burden of knowledge on how to properly use these services squarely on us engineers. As a result, we find ourselves having to constantly stay up to date and learn about all the service offerings or new changes. This takes a significant amount of time and mental energy. As developers, it can be difficult, time-consuming, and frustrating to use the AWS CLI to make 5 different API calls to describe, as an example, an AWS ECS cluster, its services, task definitions, tasks, container definitions, etc. We often find ourselves lost in documentation and having to use half a dozen of APIs to get answers to questions like "What exactly is running in this VPC?"
This means that AWS, Azure, and GCP can feel overwhelming quickly even to seasoned cloud architects. While the CSPs are fantastic at building the actual services that power our businesses, not a lot of headway has been into simplifying the day-to-day UX of querying these hundreds of services in a sane manner.
New solutions like the Cloud Control API for AWS have attempted to create a standardized interface for querying many different types of AWS resources. Unfortunately, the Cloud Control API's usage is severely limited, and users still need to know how to correctly query their data. This means more time spent reading documentation and understanding how services work and are related to one another.
While the modularity of the CSP APIs is a great logical organization system and does make sense, it's a burden on end-users in terms of the cognitive overhead and learning curve. Having to remember how hundreds of constantly changing services work and are connected leads to a caffeine addiction and time wasted playing detective.
Wouldn't it be great if we as DevOps/Cloud engineers had a simpler way to get our data out of AWS, Azure, GCP and the others? One that reflected our need to easily query any data about any service in any account without having to spend hours on docs or stack overflow?
It is for these reasons that we built CloudGraph, the GraphQL API for everything cloud. CloudGraph extracts, normalizes, processes, and enriches your cloud data allowing you to access deep insights across multiple providers effortlessly. Check out our blog post The GraphQL API for everything to learn more.
Note that CloudGraph requires READ ONLY permissions to run and as such can never mutate your actual cloud infrastructure. Additionally, none of your cloud environment information is ever sent to or shared with CloudGraph, AutoCloud, or any other third parties.
Under the hood, CloudGraph reaches out to your cloud provider(s), sucks up all of the configuration data, processes it, and stores a copy of this data for you in Dgraph. It then exposes an endpoint at http://localhost:8997 that allows you to write GraphQL Queries against your stored data. These queries not only allow you do to anything that you would do with say, the AWS SDK/CLI, but they also allow you to run much more powerful queries as well. CloudGraph ships with pre-packaged GraphQL query tools including GraphQL Playground and Altair but you can also feel free to use your own. It also includes a schema visualization tool called Voyager so you can understand relationships between entities.
CloudGraph currently supports AWS, Azure, GCP, K8s, and Tencent (several others coming soon). CloudGraph needs read permissions in order to ingest your data. To keep things easy you can use the same permissions that we use internally when we run CloudGraph to power AutoCloud. Here are the auth guides and details for how to generate credentials for each provider (feel free to leave out AutoCloud specific configuration):
System Requirements
There are 2 ways to install the CloudGraph CLI
You can install CloudGraph using homebrew with the following command:
brew install cloudgraphdev/tap/cg
Use this command to install and update CloudGraph to the latest version.
npm i -g @cloudgraph/cli
You can then add the providers you want (links to provider repos: AWS, Azure, GCP, K8s, Tencent Docs):
cg init aws cg init azure cg init gcp cg init k8s cg init tencent
You can also add as many as you want all at once
cg init aws azure gcp k8s tencent
And add in compliance policy packs to supplement your data with instant security insights:
cg policy add gcp-cis-1.2.0 cg policy add azure-cis-1.3.1 cg policy add aws-cis-1.2.0 cg policy add aws-cis-1.3.0 cg policy add aws-cis-1.2.0 cg policy add aws-pci-dss-3.2.1 cg policy add aws-nist-800-53-rev4
You can find a list of currently supported policy packs in the Policy Packs repo
You can get up and running with three simple commands:
cg init
cg launch
cg launch convenience command.
Note that if you do not want to use this command, for example, if you want to launch the Dgraph container in interactive mode, you can use the docker command below.
docker run -it -p 8995:5080 -p 8996:6080 -p 8997:8080 -p 8998:9080 -p 8999:8000
--label cloudgraph-cli-dgraph-standalone -v ~/dgraph:/dgraph --name dgraph dgraph/standalone:v21.03.1cg scan
cg scan command runs, these are usually caused by permissions issues. That said, if you encounter any problematic errors running CloudGraph you can prepend CG_DEBUG=5 to the beginning of your command as in, CG_DEBUG=5 cg scan. This will print out the verbose logs with more information and save the output to cg-debug.log. Please share your logs with us either by opening an issue on GitHub or let us know in our Slack Workspace.
That's it, you are all set to start querying! The query tool you selected during the cg init command will then be opened in your preferred browser to run queries, mutations, and visualizations on all of your cloud infrastructure! Note that if you installed any policy packs, such as AWS CIS 1.2, policy pack insight data will be automatically added to your cloud data!
Note that you may also use any GraphQL query tool you would like by connecting it to http://localhost:8997/graphql.
To stop the Dgraph instance(stop the dgraph container) run:
cg teardown
Additionally if you wish to remove the container after stopping it, run:
cg teardown --delete-image
CloudGraph stores as many previous versions of your data as you configured in the cg init command. In order to load and query a previous version of your data simply run the cg load command and select the version of your data you wish to inspect like so:
You can find the list of services currently supported for each provider in the following provider repos:
Link to full documentation: https://docs.cloudgraph.dev/overview.
To use CloudGraph, you will need to be familiar with GraphQL. This section contains a handful of example queries to get you up and running but is by no means exhaustive. If you can dream it up, you can query it! Note that you can find hundreds of additional example queries in the documentation.
Note: this section will focus on AWS, but the same ideas apply other provider like Azure and GCP
To explain how CloudGraph works consider the following query that you can run to get the ID and ARN of a single EC2 instance. Note that for the purposes of these examples we will just request the IDs and ARNs of AWS resources to keep things terse, but you can query whatever attributes you want:
query { getawsEc2( arn: "arn:aws:ec2:us-east-1:123445678997:instance/i-12345567889012234" ) { id arn } }
This query will return a JSON payload that looks like this. All of the following examples will follow suit:
{
"data": {
"getawsEc2": {
"id": "i-12345567889012234",
"arn": "arn:aws:ec2:us-east-1:123445678997:instance/i-12345567889012234"
}
},
"extensions": {
"touched_uids": 4
}
}Get the ID and ARN of each EC2 in all the AWS accounts you have scanned:
query { queryawsEc2 { id arn } }
Get the ID and ARN of all EC2 instances in one of your AWS accounts by filtering the accountId:
query { queryawsEc2(filter: { accountId: { eq: "123456" } }) { id arn } }
Get the ID and ARN of each EC2 in "us-east-1" using a regex to search the ARN:
query { queryawsEc2(filter: { arn: { regexp: "/.*us-east-1.*/" } }) { id arn } }
Do the same thing but checking to see that the region is equal to "us-east-1" instead of using a regex:
query { queryawsEc2(filter: { region: { eq: "us-east-1" } }) { id arn } }
Do the same thing but checking to see that the region contains "us-east-1" in the name instead of using eq:
query { queryawsEc2(filter: { region: { in: "us-east-1" } }) { id arn } }
Get the ID and ARN of each M5 series EC2 instance in "us-east-1"
query { queryawsEc2( filter: { region: { eq: "us-east-1" }, instanceType: { regexp: "/^m5a*/" } } ) { id arn } }
Do the same thing but skip the first found result (i.e. offset: 1) and then only return the first two results after that (i.e. first: 2) and order those results by AZ in ascending order (order: { asc: availabilityZone }) so that instance(s) in "us-east-1a" are returned at the top of the list.
query { queryawsEc2( filter: { region: { eq: "us-east-1" }, instanceType: { regexp: "/^m5a*/" } } order: { asc: availabilityZone } first: 2 offset: 1 ) { id arn } }
Do the same thing but also include the EBS Volume that is the boot disk for each EC2 instance:
query { queryawsEc2( filter: { region: { eq: "us-east-1" }, instanceType: { regexp: "/^m5a*/" } } order: { asc: availabilityZone } first: 2 offset: 1 ) { id arn ebs(filter: { isBootDisk: true }, first: 1) { id arn isBootDisk } } }
Do the same thing, but also include the SGs and ALBs for each EC2. For the ALBs, get the EC2s that they are connected to along with the ID and ARN of each found EC2 instance (i.e. a circular query).
query { queryawsEc2( filter: { region: { eq: "us-east-1" }, instanceType: { regexp: "/^m5a*/" } } order: { asc: availabilityZone } first: 2 offset: 1 ) { id arn ebs(filter: { isBootDisk: true }, first: 1) { id arn isBootDisk } securityGroups { id arn } alb { id arn ec2Instance { id arn } } } }
Get each VPC, the ALBs and Lambdas in that VPC, and then a bunch of nested sub-data as well. Also get each S3 Bucket in us-east-1. Also get the SQS queue with an ARN of arn:aws:sqs:us-east-1:8499274828484:autocloud.fifo and check the approximateNumberOfMessages. You get the idea, CloudGraph is extremely powerful.
query { queryawsVpc { id arn alb { id arn ec2Instance { id arn ebs(filter: { isBootDisk: true }) { id arn } } } lambda { id arn kms { id arn } } } queryawsS3(filter: { region: { eq: "us-east-1" } }) { id arn } getawsSqs(arn: "arn:aws:sqs:us-east-1:8499274828484:autocloud.fifo") { approximateNumberOfMessages } }
CloudGraph Policy Packs guarantee compliance across existing infrastructure for a given cloud provider. Packs are based on sets of rules/benchmarks provided by security organizations like the Center for Internet Security with the objective of keeping your infrastructure up-to-date with industry security standards. Once you have added a policy pack using the cg policy add command (i.e. cg policy add aws-cis-1.2.0) each time you run a scan CloudGraph will automatically execute your configured policies. Those results will be stored at Dgraph and linked to your existing resources, making it easy to query your compliance results alongside your resources.
For more information on currently available policy packs please visit our Policy Packs repo
Use the CloudGraph Policy Pack for AWS CIS 1.2 to query all of your CIS findings for all of your AWS Accounts:
query { queryawsCISFindings { id resourceId result rule { id description severity } } }
If you want to query several different compliance findings for a given provider like AWS at once, you can request them like this:
query { queryawsFindings { CISFindings { id resourceId result rule { id description severity } } AutoCloudFindings { id resourceId result rule { id description severity } } } }
For each CIS rule, get the resources that the rule is associated with, in this case we are quering IAM user's data to see which pass and fail:
query { queryawsCISFindings { id resourceId result rule { id description severity } iamUser { id arn name } } }
If you wanted to understand the CIS rules that apply to a particular IAM User you could use the following query:
query { getawsIamUser(id: "123456789") { name CISFindings { id resourceId result rule { id description severity } } } }
Even if you don't have any policy packs installed, you can still write powerful security queries like this to find all the unencrypted EBS Volumes:
query { queryawsEbs(filter: { encrypted: false }) { id arn availabilityZone encrypted } }
Find all the public S3 Buckets:
query { queryawsS3(filter: { access: { eq: "Public" } }) { id arn access } }
Find all the S3 Buckets that are themselves public or that can have Objects that are public in them:
query { queryawsS3(filter: { not: { access: { eq: "Private" } } }) { id arn access } }
Find all the KMS keys in "us-east-1":
query { queryawsKms(filter: { arn: { regexp: "/.*us-east-1.*/" } }) { id arn description keyRotationEnabled tags { key value } } }
Find all the burstable T series instances:
query { queryawsEc2(filter: { instanceType: { regexp: "/^t.*/" } }) { id arn availabilityZone instanceType } }
Find the default VPCs:
query { queryawsVpc(filter: { defaultVpc: true }) { id arn defaultVpc state } }
Find the public ALBs:
query { queryawsAlb(filter: { scheme: { eq: "internet-facing" } }) { id arn dnsName createdAt tags { key value } } }
Find all of the EC2s, Lambdas, and VPCs that have a Tag value of "Production":
query { queryawsTag(filter: { value: { eq: "Production" } }) { key value ec2Instance { id arn } lambda { id arn } vpc { id arn } } }
Do the same thing but look for both a key and a value:
query { queryawsTag( filter: { key: { eq: "Environment" }, value: { eq: "Production" } } ) { key value ec2Instance { id arn } lambda { id arn } vpc { id arn } } }
Do the same thing using getawsTag instead of queryawsTag. Note that when searching for tags using getawsTag your must specify both the key and value as the id like is done below with "Environment:Production":
query { getawsTag(id: "Environment:Production") { key value ec2Instance { id arn } lambda { id arn } vpc { id arn } } }
Note that billing data is currently only available for AWS. In order to successfully ingest FinOps related data you must have the Cost Explorer API enabled in your AWS Account. You can view how to do that here
Get the total cost of your AWS Account for the last 30 days, the total cost of your AWS Account month to date, a breakdown of each service and its cost for the last 30 days, and a breakdown of each service and its cost month to date as well as the monthly and month to date average costs:
query { queryawsBilling { totalCostLast30Days { cost currency formattedCost } totalCostMonthToDate { cost currency formattedCost } monthToDate { name cost currency formattedCost } last30Days { name cost currency formattedCost } monthToDateDailyAverage { name cost currency formattedCost } last30DaysDailyAverage { name cost currency formattedCost } } }
This query will return a JSON payload that looks like this:
{
"data": {
"queryawsBilling": [
{
"totalCostLast30Days": {
"cost": 7088.87,
"currency": "USD",
"formattedCost": "$7088.87"
},
"totalCostMonthToDate": {
"cost": 7089.28,
"currency": "USD",
"formattedCost": "$7089.28"
},
"monthToDate": [
{
"name": "Amazon Relational Database Service",
"cost": 548.68,
"currency": "USD",
"formattedCost": "$548.68"
},
{
"name": "Amazon Managed Streaming for Apache Kafka",
"cost": 67.49,
"currency": "USD",
"formattedCost": "$67.49"
},
{
"name": "Amazon OpenSearch Service",
"cost": 1155.04,
"currency": "USD",
"formattedCost": "$1155.04"
}
...More Services
],
"last30Days": [
{
"name": "AWS Step Functions",
"cost": 330.20,
"currency": "USD",
"formattedCost": "$330.20"
},
{
"name": "Amazon Elastic Container Service for Kubernetes",
"cost": 194.40,
"currency": "USD",
"formattedCost": "$194.40"
},
{
"name": "AmazonCloudWatch",
"cost": 310.54,
"currency": "USD",
"formattedCost": "$310.54"
}
...More Services
],
"monthToDateDailyAverage": [
{
"name": "Amazon Relational Database Service",
"cost": 54.86,
"currency": "USD",
"formattedCost": "$54.86"
},
{
"name": "Amazon Managed Streaming for Apache Kafka",
"cost": 6.74,
"currency": "USD",
"formattedCost": "$6.74"
},
{
"name": "Amazon OpenSearch Service",
"cost": 115.50,
"currency": "USD",
"formattedCost": "$115.50"
}
...More Services
],
"last30DaysDailyAverage": [
{
"name": "AWS Step Functions",
"cost": 33.01,
"currency": "USD",
"formattedCost": "$33.01"
},
{
"name": "Amazon Elastic Container Service for Kubernetes",
"cost": 19.44,
"currency": "USD",
"formattedCost": "$19.44"
},
{
"name": "AmazonCloudWatch",
"cost": 31.05,
"currency": "USD",
"formattedCost": "$31.05"
}
...More Services
],
}
]
},
"extensions": {
"touched_uids": 212
}
}Get each EC2 instance in your AWS account along with its daily cost:
query { queryawsEc2 { arn dailyCost { cost currency formattedCost } } }
This query will return a JSON payload that looks like this. All of the following examples will follow suit:
{
{
"data": {
"queryawsEc2": [
{
"arn": "arn:aws:ec2:us-east-1:12345678910:instance/i-0c8b3vhfgf8df923f",
"dailyCost": {
"cost": 2.06,
"currency": "USD",
"formattedCost": "$2.06"
}
},
{
"arn": "arn:aws:ec2:us-east-1:12345678910:instance/i-060b3dsfds7sdf62e3",
"dailyCost": {
"cost": 2.06,
"currency": "USD",
"formattedCost": "$2.06"
}
},
...More EC2 Instances
]
},
"extensions": {
"touched_uids": 28
}
}Get each NAT Gateway in your AWS account along with its daily cost:
query { queryawsNatGateway { arn dailyCost { cost currency formattedCost } } }
CloudGraph ingests your CloudWatch Metric data and stores it along with select AWS services. This feature is currently in beta and will work for EC2 only:
query { queryawsEc2 { arn cloudWatchMetricData { lastWeek { cpuUtilizationAverage networkInAverage networkOutAverage networkPacketsInAverage networkPacketsOutAverage statusCheckFailedSum statusCheckFailedInstanceSum statusCheckFailedSystemSum diskReadOpsAverage diskWriteOpsAverage diskReadBytesAverage diskWriteBytesAverage } lastMonth { cpuUtilizationAverage networkInAverage networkOutAverage networkPacketsInAverage networkPacketsOutAverage statusCheckFailedSum statusCheckFailedInstanceSum statusCheckFailedSystemSum diskReadOpsAverage diskWriteOpsAverage diskReadBytesAverage diskWriteBytesAverage } last6Hours { cpuUtilizationAverage networkInAverage networkOutAverage networkPacketsInAverage networkPacketsOutAverage statusCheckFailedSum statusCheckFailedInstanceSum statusCheckFailedSystemSum diskReadOpsAverage diskWriteOpsAverage diskReadBytesAverage diskWriteBytesAverage } last24Hours { cpuUtilizationAverage networkInAverage networkOutAverage networkPacketsInAverage networkPacketsOutAverage statusCheckFailedSum statusCheckFailedInstanceSum statusCheckFailedSystemSum diskReadOpsAverage diskWriteOpsAverage diskReadBytesAverage diskWriteBytesAverage } } } }
When you think, "in terms of a graph", you can do almost anything with CloudGraph. Say for example that you want to know what Lamba functions don't belong to a VPC (i.e. they don't leverage VPC networking). Because CloudGraph connects all resources that have relationships, such as VPC parents to their Lambda children, you are able to answer this question easily. Simply check to see what lambda functions the VPC is "connected" to, and compare that against the list of all lambda functions like so:
query { queryawsVpc { id arn lambda { id arn } } queryawsLambda { id arn } }
Today, the biggest limitation with CloudGraph and our query abilities is we don't support nested filtering based on child attributes. So for example, as cool as it would be to do the following, it's just not possible yet:
query { # This won't work just yet... queryawsEc2(filter: { ebs: { isBootDisk: true } }) { id arn ebs { id arn } } # So you have to do this instead :( queryawsEc2 { id arn ebs(filter: { isBootDisk: true }) { id arn } } }
This is actually not a limitation of CloudGraph, but rather a feature that still needs to be implemented with Dgraph. You can view and comment on the discussion thread here
CloudGraph ships with 2 awesome query tools and a GraphQL schema explorer. Remember, you can use ANY GraphQL query tool if you would prefer another option, just connect it to your exposed /graphql endpoint!
GraphQL playground has a fluid and engaging UX that is great for querying a GraphQL schema quickly and simply. It has built-in automatically generated documentation and auto-completion while you type. To access playground, either select it as your preferred query tool in the init command OR visit /playground in the server CG spins up.
Altair is another great GraphQL query tool that packs a ton of features for power users. Do things like autocomplete queries, dynamically add fragments, and export/import collections of queries. To access Altair, either select it as your preferred query tool in the init command OR visit /altair in the server CG spins up.
GraphQL Voyager is an awesome way to explore the schema(s) for your CG providers. It gives you a great bidirectional chart containing all your types and queries. You can click entities or arrows to discover connections, search for something specific, and get a deeper understanding of your schema. To access voyager, visit /voyager in the server CG spins up.
Comments, questions, or feedback? Please Join Our Slack Workspace we would love to hear from you.
If you're interested in contributing to CloudGraph please check out our Contribution Guidelines.
You can either run CloudGraph locally, or you can deploy it to your cloud provider of choice. Terraform modules and guides for cloud deployments are coming soon!
Interested in a fully managed SaaS/self hosted version of CloudGraph that has built in 3D visualization capabilities, automated scans, and hundreds of additional compliance checks? Check out AutoCloud for more details.
If you encounter any errors running CloudGraph you can prep