Creating a Lookup Table
In order to support the big data processing requirements that some organizations have, it may be more practical or efficient for you to compute carbon metrics within your existing processing. To do so, we support the generation of a lookup table that can be utilized as an additional step in your pipeline.
The lookup table maps the estimated energy (kilowatt-hours) and carbon emissions (CO2e) to 1 unit of usage, for all the unique combinations of region, service name, usage type and usage unit in the billing data of your cloud provider(s).
Once generated, this lookup table (CSV file) can be deployed to your ETL or other data processing pipeline. Then when processing your billing data, you can simply multiply your usage amount by the values in the lookup tables to estimate energy and CO2e. This approach avoids having to use the Cloud Carbon Footprint application code directly, and works regardless of the programming language or environment used in your pipeline.
To generate this lookup table:
- Make sure you have a CSV file inside the
cli
package, that contains all the unique region, service name, usage type and usage unit variations in your billing data, along with the vCPUs for that line item, if it exists. You can see an example of this using AWS data here, and below for example queries to create this file. - Run the following:
The options for this command are:
We would like to thank @mfulleratlassian for contributing this functionality.
#
Example queries to create input CSV file#
AWS - Athena Query#
GCP - BigQuery Query#
Azure - Yarn ScriptCreating an input file for Azure using billing data requires the use of the Consumption Management API rather than a direct query.
To assist with this, we have created a script that makes use of your configured credentials in the packages/cli/.env
file to query and output the needed mappings to a CSV file.
To use this script, run the following yarn command with the provided parameters:
The options for this command are: