AWS Integration

Overview

Cost Management retrieves asset prices from the public AWS pricing API by default. You can leverage Cost Profile to interface directly with your account to receive accurate price data from AWS. This integration will properly account for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, spot usage, and more.

Necessary permissions are required to create the Cost and Usage Report (CUR), and to add IAM credentials for Athena and S3. The ability to add and use CloudFormation templates is an optional permission.

Users must perform the below steps and complete a few setups to create a Cost profile in the controller.

Step 1: CUR Setup

• Refer CUR Creation for creating Cost and Usage Reports. It is mandatory to enable Resource IDs and Athena integration when creating the CUR. Note down the bucket name that you create for CUR data.

• If the Billing and Cost Management page cannot be accessed, the owner of the organization’s root account must follow the instructions provided here.

Step 2: Athena Setup

In addition, as a part of CUR creation, Amazon creates a CloudFormation template that is used to create the Athena integration. This template is created in the CUR S3 bucket under user-billing-prefix/curname and typically has the filename crawler-cfn.yml. Deploy this CloudFormation template to complete the CUR Athena integration. Refer here to know more details on CloudFormation templates.

Once the Athena setup is complete with the CUR, create a new S3 bucket for Athena query results.

• Navigate to AWS S3 and select Create Bucket
• Choose a name that adheres to the format aws-athena-query-results- and make sure to use the same region as the one that was used for the CUR bucket
• Select Create Bucket
• Navigate to AWS Athena and select Settings
• Set Query result location to the S3 bucket that got created now.

Step 3: Required Policies for Role/User Integration

Attach the below two policies to the same role or user. Pick a user to integrate via service key (or) a role to integrate via IAM annotation. The SpotDataAccess policy statement is optional if the spot data feed is configured.

Policy 1

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "AthenaAccess",
         "Effect": "Allow",
         "Action": [
            "athena:*"
         ],
         "Resource": [
            "*"
         ]
      },
      {
         "Sid": "ReadAccessToAthenaCurDataViaGlue",
         "Effect": "Allow",
         "Action": [
            "glue:GetDatabase*",
            "glue:GetTable*",
            "glue:GetPartition*",
            "glue:GetUserDefinedFunction",
            "glue:BatchGetPartition"
         ],
         "Resource": [
            "arn:aws:glue:*:*:catalog",
            "arn:aws:glue:*:*:database/athenacurcfn*",
            "arn:aws:glue:*:*:table/athenacurcfn*/*"
         ]
      },
      {
         "Sid": "AthenaQueryResultsOutput",
         "Effect": "Allow",
         "Action": [
            "s3:GetBucketLocation",
            "s3:GetObject",
            "s3:ListBucket",
            "s3:ListBucketMultipartUploads",
            "s3:ListMultipartUploadParts",
            "s3:AbortMultipartUpload",
            "s3:CreateBucket",
            "s3:PutObject"
         ],
         "Resource": [
            "arn:aws:s3:::aws-athena-query-results-*"
         ]
      },
      {
         "Sid": "S3ReadAccessToAwsBillingData",
         "Effect": "Allow",
         "Action": [
            "s3:Get*",
            "s3:List*"
         ],
         "Resource": [
            "arn:aws:s3:::${AthenaCURBucket}*"
         ]
      }
   ]
}

Policy 2

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "SpotDataAccess",
         "Effect": "Allow",
         "Action": [
            "s3:ListAllMyBuckets",
            "s3:ListBucket",
            "s3:HeadBucket",
            "s3:HeadObject",
            "s3:List*",
            "s3:Get*"
         ],
         "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
      }
   ]
}

Kubernetes clusters run in different accounts

Perform the following steps:

• On each sub account that runs Cost Management, attach the below 2 sub account policies to the same role or user. If selecting "Use Cluster Credentials", update the two sub account policies below for the role or user used in Cluster Provisioning and skip 'Step 4'. The SpotDataAccess policy statement is optional if the spot data feed is configured.

Sub account policy for CUR Integration

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "AssumeRoleInMasterPayer",
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Resource": "arn:aws:iam::${MasterPayerAccountID}:role/costmanagementRole-${This-account’s-id}"
      }
   ]
}

Sub account policy for Spot Integration (optional)

Master Account Policy

Attach this policy to a role on the master payer account (replace ${AthenaCURBucket} variable)

Now, you must add the following trust statement to the role the policy is attached to (replace ${costmanagementClusterID} variable):

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::${AWSAccountID}:root"
         },
         "Action": [
            "sts:AssumeRole"
         ]
      }
   ]
}

Step 4: Attach IAM permissions to Cost Management

If Use Cluster Credential is selected during the creation of the Cost Profile, this Step 4 does not need to be performed.

Once the policies are created, users can attach those policies to Cost Management. The following methods are supported:

Via Access Key

• Login to AWS IAM Access page
• Navigate to Access Management -> Users
• Find the Cost Management User and select Security Credentials -> Create Access Key. Note the Access key ID and Secret access key. You will use it to either Create a secret from helm values or Create and use an existing secret.
• Attach this Access ID and Secret key, and create a Cost Management AWS Cloud Credential

Via Role ARN

• Create an OIDC provider for your cluster
• Create a Role
• Attach the policies created in Step 3
• Attach this Role ARN when creating a cloud credentials of type Cost Management and provider AWS

Step 5: Configure CUR config values to Cost Management

• Athena Bucket Name: An S3 bucket to store Athena query results to which Cost Management has permission to access.

• The name of the bucket should match s3://aws-athena-query-results-*, so the IAM roles defined in Step 3 will automatically allow access to it.

• The bucket can have a Canned ACL of Private or other permissions.

• Athena Region: The AWS region Athena is running in.

• Athena Database: The name of the database created by the Athena setup.

• The Athena database name is available as the value (physical id) of AWSCURDatabase in the CloudFormation stack created in Step 2.

• Athena Table: The name of the table created by the Athena setup.

• The table name is typically the database name with the leading athenacurcfn_ removed (but is not available as a CloudFormation stack resource).

• Master Payer Account ARN: If using a multi-account setup, it is mandatory to set to the Amazon Resource Number (ARN) of the role in the master payer account. For example, arn:aws:iam::530337586275:role/costmanagementRole.

Spot Data feed integration

Cost Management reconciles the users' spot prices with CUR billing reports as they become available (usually 1-2 days), but pricing data can be retrieved hourly by integrating directly with the AWS spot feed. To enable, follow the steps provided here.