Understanding and managing Cloud Expenditure is an increasingly sought after topic with companies as the focus on Cloud Financial Management is increasing. Existing AWS customers with migrated workloads in the Cloud that are mature are looking to optimise their Cloud Costs and realise the dream of low-cost elasticity.

At the same time, new projects that start their journey in the Cloud can also find themselves out of control of their Cloud Expenditure, without being able to control and react to unexpected spending patterns. The main offering the Cloud, elasticity, that enables easy access to infrastructure that would otherwise take a long time to be provisioned, also provide an easy way accidentally end up with a large bill. A slight misconfiguration of the permission policies could allow a new developer to unintentionally rack up recurring Cloud costs.

Managing this in the long term requires a focus on technical guardrails, processes, and culture of cost awareness. This organisational maturity is not achieved on a single day and should be a target for a long-term transformation across several teams. FinOps Foundation defines a maturity journey in their FinOps Framework for this exact reason.

When it comes to AWS specifically, depending on the size of the AWS Organization, this maturity journey can take form of a months-long effort. However, as a foundation there are three basic AWS services that can be used with very little effort to gain visibility into the Cloud Expenditure and build a more robust reactive capability to address unexpected spending patterns before they get out of hand. By doing so, an organisation that is starting their FinOps maturity journey for the AWS deployments, can build a strong Inform phase of visibility and allocation , so that the Optimise and Operate phases can be implemented on strong foundations.

The following content assumes that AWS Cost Explorer along with Consolidated Billing is enabled on the AWS Organization Management Account. Cost Explorer helps to track costs, perform Root Cause Analysis on Cloud Costs, and generate periodical reports. The following services enables performing these tasks in a more meaningful manner by,

1. Setting up a cost allocation strategy for the overall AWS Organization (FinOps Allocation Capability)
2. Establishing visibility into spending patterns and proactive capability to unexpected expenditures (FinOps Budgeting Capability)
3. Enabling automated anomaly detection for Cloud Cost (FinOps Anomaly Management Capability)

Cost Allocation Tagging

AWS Tags are key value pairs that can be used to assign metadata to different resources. Tags enable an additional dimensions of data points to be attached to make meaning out of potentially a vast number of resources that will be spawned in the entire AWS Organization.

When used as a standardised tagging framework, a pre-defined set of tags can be used as Cost Allocation Tags, as classes of cost expenditure in the Organization. These can then be used for later cost analysis and as a platform to build a shareback or chargeback model of consolidated AWS cost down the line. AWS Cost Explorer, AWS Budgets, and Cost Anomaly Detection has first class support for Cost Allocation Tags, so enabling and using them form the foundation of cost allocation FinOps Capability.

AWS also generates some Cost Allocation Tags for resources that can be activated to be allocated based on the context of the resource creation. For an example, the tag aws:createdBy that can be activated in the Cost Allocation Tags console to be applied to resources to track the AWS principals involved in creating them.

A major part of implementing a tag based cost allocation strategy is to ensure that the appropriate tags are being applied to resources when they are created. This can be enforced by using AWS Organization Tag Policies where necessary Tags are marked as enforced to prevent non-compliant resources from being launched. It should be noted that enforcement of a Tag Policy will have direct impact on the end user experience for developers, devops, and other engineers who interact with AWS. Effective rollout of a tagging policy should be closely aligned with proper communication and process changes to make sure every stakeholder is aware of the enforced changes.

AWS Resource Groups and Tag Editor Service can be used to monitor for tagging compliance across the Organization if a soft enforcement approach is needed.

In addition to building a basis for cost allocation, a Cost Allocating Tagging strategy will also help build an overall cost-aware culture in the AWS Organization users, especially if the users who spawn resources are distributed owners of different Organization Units (OU) in the AWS Organization.

AWS Budgets

AWS Budgets is a mature Service that provides the capability to specify soft and hard limits on AWS Cloud Spending to raise notifications when those limits are exceeded. This is the perfect AWS Service to implement FinOps Budgeting capability to build a proactive approach to addressing Cloud Cost Expenditure. This can be done by

1. Specifying a budget for part of or the entire AWS usage
2. Specify actual and forecasted Cloud spending thresholds to generate alerts to
3. Optionally, specify automated actions to trigger when those thresholds are met

The main business offers of AWS Budgets as a Service are,

1. Monitor cost usage on multiple dimensions and granularity and generate Alerts
2. Generate scheduled reports
3. Respond to generated Alerts through attached Actions

Budgets can be defined for various types of cost expenditures. These include,

1. Cost budgets – actual cloud costs as defined by a dollar amount
2. Usage budgets – usage of specific AWS services
3. Reservation budgets – monitoring utilisation and coverage of Reserved Instance pricing resources
4. Savings Plan budgets – monitoring utilisation and coverage of resources that can be included in AWS Savings Plans

Additionally, these Budgets can be defined as periodic recurring budget on a daily, monthly, quarterly, or yearly cadence, or as a fixed expiring budget that does not reset.

When it comes to the granularity of monitoring for budget exhaustion, AWS Budgets offer a number of different dimensions. Budgets can be for the overall AWS Organization, specific Linked Accounts, services, instances types, tags, usage types, or even for specific API operations (among other dimensions). This enables organisations to monitor for part of the AWS cost that they are mostly focusing on.

In addition to being able to react to unexpected cost expenditures quickly, using AWS Budgets also help during Optimize phases of FinOps journey, as granular budgets can expose spending patterns in areas of the Cloud Architecture that is not expected during design.

AWS Cost Anomaly Detection

Cost Anomaly Detection is an AWS Service to automatically detect anomalous spending patterns, including sudden increases in the expected AWS Bill out of the usual spending. AWS uses a multi-layered Machine Learning approach to learn the history of the AWS spending for the specific AWS Organization to then detect outliers in the patterns. While AWS Budgets help set up spending thresholds manually, AWS Cost Anomaly Detection fills the gap of identifying abnormal spending patterns with automatic thresholds as learnt from past usage. This is a good Service to use to establish Anomaly Management Capability of FinOps.

Like AWS Budgets, Cost Anomaly Detectors can be set up for various dimensions of the AWS Organization. This could be for AWS Services, Linked Accounts, specific Cost Allocation Tags, or specific Cost Categories.

When an anomalous pattern is detected, which is usually done as soon as cost data is consolidated in Cost Explorer, AWS tries to identify a best estimate of the largest contributor of the spending pattern. This is then marked as the Root Cause and presented to the user for Root Causes Analysis that can be done on the Cost Explorer Web Console. In some cases, there could be multiple root causes for the anomalous spending pattern, where AWS will elect the top most possible two causes for RCA.

New Cost Explorer users (Payer or Regular AWS Accounts that enabled Cost Explorer since March 2023) get a default AWS Services Monitor with a daily alert email at no cost. This is to set up new AWS users with standard best practices to minimise unexpected spending from getting out of hand.

Bonus: AWS Cost Categories

AWS Cost Categories is a feature of Billing and Cost Management, where a more complex cost allocation method can be established on top of Cost Allocation Tags. Cost Categories allow defining rules that collect different types of cost sources to classes of cost expenditure. These sources can be

1. AWS Account
2. Charge Type
3. AWS Service
4. Region
5. Usage Type
6. Cost Allocation Tags
7. Other Cost Categories

What Cost Categories enables is a detailed allocation of granular costs to different classes that would represent different participants in the overall AWS bill. These classes can be different teams, departments, or projects that can be included a shareback or chargeback approach. Cost Categories also allow splitting certain charges by proportion or on a fixed amount, which further allows dividing cost components among different units within the organisation. By doing so, users can address minute details in AWS spending that cannot be zeroed in with just using Cost Allocation Tags.

Cost Categories have first class support in Cost Explorer, Cost Anomaly Detection, and AWS Budgets.

Conclusion

FinOps maturity takes many iterations of the FinOps phases being implemented to build the Capabilities in the Cloud. The above basic AWS Services can be utilised with minimum effort to start the process, wherever the user is in the maturity journey. They are built with these Capabilities in mind, and helps the users to establish best practices in Cost Management in AWS.