In the same way applications run on your laptop, accessing anything on the internet also requires those applications to run on a computer. These computers are called servers. They are just like a laptop but do not have a screen or keyboard and must be located somewhere where they have access to the internet, power, and cooling. Such places are called data centers. These facilities can range in size from small 100ft2 cabinets up to massive 400,000ft2 “hyperscale” warehouses (Shehabi et al, 2016). Whenever you use any service on the internet, you are connecting to one of many millions of servers located in one of many thousands of data centers around the world.
Data centers are responsible for between 1-1.5% of global electricity usage (Andrae and Edler, 2015), approximately 200,000 GWh, which amounts to 0.3% of human generated CO2 emissions (Jones, 2018). Although there are several elements which make up the energy footprint of a data center, electricity is the largest, split across four main components. This post will consider those key components before examining the changes brought by cloud computing and what factors may affect future data center energy profiles.
Having grown from around 11 million in 2006, as of 2020 there are an estimated 18 million servers deployed in data centers globally (Shehabi et al, 2016). Power drawn is partially related to usage, expressed as a percentage of maximum power. This is related to the number of Central Processing Unit (CPU) sockets and has remained static since 2007: 118W for single socket servers and 365W for two socket servers (Shehabi et al, 2016; Shehabi et al, 2018).
Power proportionality is key to understanding the efficiency of servers. This scales in proportion to utilisation. With perfect power proportionality, a server at 10% utilisation will draw 10% of its maximum power (Shehabi et al, 2016). This is measured as Dynamic Range – a ratio between idle power and maximum power which can be affected by hardware properties, power management software and the server configuration (Shehabi et al, 2016).
These improvements in server dynamic range have been coupled with improvements in utilisation caused by software management systems and the move to hyperscale facilities run by the cloud providers. However, despite this most servers still rarely run at full utilisation – the most efficient run at only 50% (Masanet et al, 2013; Shehabi et al, 2016). This translates to 40,000 GWh/yr in direct server electricity consumption in the US as of 2020, of which half can be wasted by idle servers (Shehabi et al, 2016).
The amount of data generated by humanity is growing every year (Statista, 2018), and it needs to be stored on disks. Power drawn per disk varies by drive type. Hard Disk Drive (HDD) wattage is not related to capacity and was estimated at 14W/disk in 2006, decreasing by 5% each year to 8.6W/disk in 2015 (Shehabi et al, 2016). Solid State Disk (SSD) wattage has remained a constant 6W/disk since 2010 but the wattage per terabyte (TB) has been improving, with capacity per watt increasing 3-4x between 2010-2020 (Shehabi et al, 2016).
Total electricity usage for disks in the US in 2020 is estimated at just over 8,000 GWh/yr across a total of 1,000 million TB of storage (Shehabi et al, 2016). With an estimated lifespan of 4.4 years, number of disks deployed is plateauing, but total capacity is in-creasing (Shehabi et al, 2016).
Servers need to be connected to each other, and to the internet; this is the network component. Network devices use power related to the number of ports and their speed (Shehabi et al, 2016).
Wildly varying estimates for the energy intensity of the internet have been published, ranging from 136 kWh/GB in 2000 to 0.004 kWh/GB in 2008, but a more recent estimate analysing calculation methodologies settled on 0.06 kWh/GB for 2015 (Aslan et al, 2018). This is decreasing by 50% every 2 years (Aslan et al, 2018).
Calculating the energy intensity of the internet is difficult – Aslan et al (2018) only considers fixed line networks in developed countries. Calculations are missing for mobile networks that will account over 20% of all internet traffic by 2022, growing at 46% per year (Cisco, 2019); and internal connectivity within data centers is not included but is doubling every 12-15 months (Singh et al, 2015). These excluded factors and no recent calculations examining networking equipment speeds up to current fastest 400Gb (Ethernet Alliance, 2019) devices means that it is difficult to estimate the true energy impact of networking today.
The data center building consists of infrastructure to support the servers, disks and networking equipment it contains. This includes cooling, power distribution, backup batteries and generators, lighting, fire suppression, and the building materials themselves (GHG Protocol, 2017). The infrastructure overhead is measured using Power Usage Effectiveness (PUE) (Uptime Institute, 2019). This is the ratio between power drawn by the infrastructure components and power delivered to the servers, disks and networking equipment (GHG Protocol, 2017).
Using the PUE ratio alone has been criticised because it will decrease when IT load increases (Brady et al, 2013) even though efficiency may not have improved. This makes it useful to compare facilities rather than indicating efficiency by itself. Although out of scope for this briefing, data centers have environmental impacts wider than just energy: water used for cooling and the life cycle of IT equipment are other important factors not included in PUE (GHG Protocol, 2017). Metrics such as Water Use Effectiveness and Land Use Effectiveness, alongside Life Cycle Analysis, have been suggested to help understand true impacts (Kass and Ravagni, 2019).
In a traditional data center, only 17.5% of electricity generated by a power plant ultimately reaches the servers – this is due to grid losses combined with losses in the power distribution systems within the data center (Zhao et al, 2014). Fuel cells have been investigated as a method of eliminating these losses (Zhao et al, 2014) and could increase efficiency to 29.5%.
With further modifications to data center design to use Direct Current (DC) from the fuel cell and bypassing the Uninterruptible Power Supply (used for backup power but not needed with gas reliability at 99.999%), 53.2% efficiency could be achieved (Zhao et al, 2014).
Physical IT equipment can be measured to determine the environmental impact in embodied energy as well as power drawn during actual usage. This can then be combined with calculations from the data center components to calculate emissions. Indeed, these calculations are part of the Greenhouse Gas Protocol (GHG Protocol, 2017) and standards exist for constructing energy efficient data centers (Huusko et al, 2012). These types of emissions fall under the Scope 1 and Scope 2 reporting guidelines (GHG Protocol, 2015) that many organisations are required to publish (Department for Business, Energy & Industrial Strategy, 2018).
However, when IT workloads are moved to the cloud and resources are purchased in tiny “virtual” units on a pay-as-you-go basis, their associated emissions shift to voluntary Scope 3 reporting as “indirect” or outsourced emissions. Data to calculate actual emissions is also no longer available because the major cloud vendors (Amazon Web Services, Google Cloud and Microsoft Azure) publish only aggregated global data, and with varying degrees of transparency.
Amazon is the least transparent – they report limited environmental data other than their 2018 total carbon footprint: 44.4 million tCO2e (Amazon, 2020). Since this includes all of Amazon’s operations and is not broken out for the Amazon Web Services cloud business, this is not a useful figure. This lack of transparency resulted in Amazon being criticised in a Greenpeace report (Cook, Jardim and Craighill, 2019).
With 53% of servers expected to be in hyperscale facilities by 2021 (Cisco, 2018) and the cloud computing market growing from $6bn in 2008 to $288bn in 2019 (Forrester, 2019), it is important that cloud vendors are transparent about their environmental footprint.
However, the cloud is not all negative. Hyperscale providers operate at such a scale that they can justify activities such as Google building their own servers (Metz, 2016; GCP, 2017) and Microsoft constructing the first ever gas data center (Belady & James, 2017), all of which contribute to improving energy efficiency.
The technology sector is also the largest purchaser of renewables (Kamiya, and Kvarnström, 2019) (but what does 100% renewable actually mean?) and in Jan 2020, Microsoft announced a GHG Protocol compliant sustainability calculator so customers can calculate their individual cloud carbon footprint (Microsoft, 2020). A similar approach should be adopted by all cloud vendors.
Efficiency challenges ahead?
Although the last 20 years have seen major efficiency improvements, predictions suggest these may be coming to an end. As a result of market growth and diminishing returns from existing approaches to efficiency improvements there is a suggestion that data center energy usage will double by 2030. If electricity continues to be a major source of data center energy and is generated from non-renewable sources, data center emissions could exceed the aviation industry which is currently responsible for 2% of annual human-generated CO2 (IATA, 2020).
Data center energy projections have been wrong in the past (Malmodin and Lundén, 2018; Jones, 2018; Masanet et al, 2019) and improvements such as fuel cell powered data centers are promising. However, if efficiencies do not continue, we could see usage grow within a range of 3-13% of global electricity by 2030 (Andrae and Edler, 2015). Several scenarios could combine to hamper future improvements:
- Moore’s Law states that CPU performance per watt doubles every 1.5 years (Bashroush, 2018). If data center hardware continues to only be refreshed every 4.4 years, then major efficiency improvements may be missed (Bashroush, 2018). This also assumes that Moore’s Law will continue, which is uncertain (Huang, 2015). Can we overcome the physical limits of ever-smaller chips? If replacement cycles become more frequent, how will that affect other environmental metrics such as embodied energy and waste?
- The introduction of new types of chips for specialist applications e.g. Graphics Processing Units (GPUs) for machine learning, could draw more power with unknown efficiency profiles (Shehabi et al, 2018). What efficiency profile will these chips have, and will we see historically similar performance per-watt improvements?
- Hyperscale data centers are often in regions with abundant access to renewable energy, such as Google’s Finland data center (Google, 2018). However, these locations tend to be away from population centers which means higher network response times as data must travel further to the end-user. As urbanisation increases, the need for low latency will require data centers to be sited closer to the user (Kass and Ravagni, 2019) but these locations may be less suitable for access to renewable sources of electricity or natural water sources for cooling. Can this be mitigated by gas fuel cells?
Data center energy usage is significant but historical efficiency improvements mean that growth has decoupled from energy consumption. Trends such as the cloud allow efficiency improvements to take place at huge scale but there are problems on the horizon. Approaching the physical limits of Moore’s Law and new technologies such as machine learning mean historical improvements cannot be assumed. The data center industry is changing rapidly and how that will affect energy profiles is uncertain.