Prepare AWS Account - Set Up Billing and Reporting

How to enable detailed billing, select Nucleator Tags for detailed billing, and link your new Account to your Master Account for Consolidated Billing

 

As your deployed footprint at AWS increases it becomes more important to understand and manage this infrastructure.  There are several powerful tools that help you to do this.  Nucleator supports these key AWS key infrastructure management tools:

Many other tools exists in this space however, these four are generally underpinning most AWS infrastructure analysis processes and Nucleator works to support and enhance all these areas.

Linked Accounts and Consolidated Billing

The use of linked accounts is a business by business decision and is not required by Nucleator.  Nucleator does support operation across multiple AWS accounts (linked or otherwise) through use of the account parameter.  Nucleator can create Cages and Stacksets across a business' set of linked accounts.  This enables Nucleator to be a powerful deployment tool across multiple lines of business within your enterprise.

It is important for tracking purposes that consolidated billing is enabled so that AWS infrastructure reporting can consolidated and analyzed.  To sign up for consolidated billing

  1. Sign in to the AWS Management Console and open the Billing and Cost Management console at https://console.aws.amazon.com/billing/home#/.
  2. Click Consolidated Billing and follow the instructions on the page.

Once the account owners have agreed to link the information for the accounts will be combined.  More information about consolidated billing can be found at http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html

User Tags

AWS allows for up to 10 user defined tags to be applied to most resources.  These user tags can be defined (by use convention) to be common between all resources for your AWS account or unique by resource.  The total number of user tags that can be defined can exceed 10 but only 10 can be applied to any one resource.

In order to maximize the utility of these user tags it is important to establish conventions within your organization.  By having a set of user tags that are common among all AWS infrastructure powerful business analyses can be performed that span all of your AWS accounts.  For example if infrastructure in support of a major client are all tagged the same then resource and cost analyses for this client can be easily viewed no matter if the resource is provisioned by marketing, R&D or sales.  Also it is valuable to have some user tags be specific to the class or resource or department.   Nucleator uses such tool-specific tags as described here:  Nucleator Resource Tagging.  However, there is no AWS policy engine that will enforce your conventions so organizational processes will be needed to ensure that user tags are properly used.

47Lining recommendations for global user tags:

Applying these tags across all of your AWS resources in addition to using linked accounts to segregate the most macro of organizational divisions will allow for detailed analysis of your AWS resources and the charges that they drive.  For this to happen all of your AWS resources will need to adopt your tagging convention, including resources that are started manually.  When a common tagging methodology is applied to all your resources understanding the spent patterns and drivers will improve your AWS spend effectiveness.  To this end it is helpful to note that Nucleator can cause the creation of an untagged resource - AWS elsastic beanstalk will create an S3 bucket in every region it is run.  This bucket will be named elasticbeanstalk-{region}-{hash} and is a common resource for all elastic beanstalk operations within your account for that region.  Since this is a shared resource, Nucleator does not have the necessary information to appropriately apply tags to it.  Account owners should apply the 4 tags above to this bucket (done by modifying the bucket properties) manually.

Note: In order for user tags to flow to detailed AWS usage reports they will need to be marked for inclusion in these reports - see detailed billing reports section below.

Detailed Billing Reports

Don't be fooled by the name – detailed billing reports provide a wealth of AWS usage information that is useful well outside of cost management.  These reports will include hour by hour usage information for all resources at AWS as well as the costs associated with this usage.  This information can provide a great deal of insight into your AWS resource usage, profile vs time, opportunities for cost-saving purchases such as reserved instances, spend by organization, tag or person etc.  

To enable detailed billing reports you will need to do the following for your master account:

Enable Detailed BIlling Reports

Icon

  • Log in to your AWS account console with billing change capability

  • Go to the S3 console and create a bucket where detailed billing information will be stored 
    1. Modify this bucket's policy to include the policy below

 

Icon

In Accounts that have been prepared for use by Nucleator, the above step is performed automatically by nucleator account setup

Nucleator creates a bucket with the following naming convention:

billing-nucleator-<account_name>-<customer_domain>-<uuidish>

where:

<account_name> is the account friendly name for the Acount

<customer_domain> is the customer domain specified in customer's config file, with any "." characters replaced by "-" characters.

<uuidish> is a distinct character string generated by Nucleator that is intended to be unique. Because it is only six characters long, uniqueness is not guaranteed but is extremely probable.

 

  • Select the "billing and cost management" from the pull down next to your "user @ account" designation in the upper right of the console page 
  • Select "Preferences" from the menu on the left
  • Select the checkbox next to "Receive Billing Reports"
  • Enter the bucket name where you would like billing reports to be located (See naming convention above if your billing bucket was created by Nucleator)
  • Click "Verify" to ensure that the policy for the bucket will allow billing reports to be written 
  • Check all four check boxes for all types of billing reports
  • "Save preferences"
  • Click "Manage Report Tags"
    1. Ensure that all the global user tags are checked (above we recommended using the tags Offering, Purpose, Owner, and Name globally)
    2. Ensure that all Nucleator tags are checked - NucleatorCage, NucleatorCustomer, NucleatorGroup, NucleatorStackset, NucleatorStacksetInstance
    3. Check any other tags that will be of general ongoing interest to your organization (selecting all is not recommended as this list is a superset of all tags defined within your account at any time and the usefulness of many of these one-off tags is zero)
  • "Save" Report Tags

 

The AWS "sample policy" for a billing report bucket at the time of this writing is:

{
 "Version": "2008-10-17",
 "Id": "Policy1335892530063",
 "Statement": [
 {
 "Sid": "Stmt1335892150622",
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::386209384616:root"
 },
 "Action": [
 "s3:GetBucketAcl",
 "s3:GetBucketPolicy"
 ],
 "Resource": "arn:aws:s3:::billing-47lining-com"
 },
 {
 "Sid": "Stmt1335892526596",
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::386209384616:root"
 },
 "Action": [
 "s3:PutObject"
 ],
 "Resource": "arn:aws:s3:::billing-47lining-com/*"
 }
 ]
}

 

Applying BI Tools

Once you have enabled detailed billing reports your billing report bucket in your master account will receive a large volume of information updated multiple times per day.  The information is organized by month and single files will include information about resource use and cost for all linked accounts.  Gaining insight into your AWS footprint will not be achieved by simply viewing these files as the amount of data is too large.  At 47Lining we store our billing information in AWS' data warehouse offering Redshift and update the database twice daily.  By attaching BI tools to the Redshift database a wealth of information can be learned.  

47Lining Nucleator has a set of tools currently in the alpha stage of development that enable the importing of detailed billing information, transformation of this information, and standard analysis views into this data.  While this offering is not yet publicly available, 47Lining is looking for alpha users to provide feedback.  If interested please contact 47Lining by email at info@47lining.com and indicate "Billing Toolset alpha program" in the subject.  The information below is based on our own internal use of this toolset.

There are a variety of BI tools that can be used to perform the analysis and presentation of your AWS usage information each with its own strengths and weaknesses.  This is a summary of the tools we have successfully used within 47lining for evaluating our data:

With a BI toolset attached to Redshift there are several classes of report or metric that can be produced.  The dashboards we most often look at by class of report are:

This is just a partial lis of the information that can be seen in the AWS detailed billing reports and if additional sources of data, such as web server log archives and platform event log databases, are added there is no limit to business valuable information that can be learned.

 

Example Metrics:

Per-Product Cost by Hour and Linked Account (Tableau):

 

Nucleator Cages by Hour and Linked Account (Looker):

 

Running ec2 Instances by Hour of the Day (tableau):

 

Next: Hardening your Nucleator Installation and Launching Nucleator's User Interface

 

 

Installation Documentation Releases License Community