I was the lead author on a short peer reviewed communication paper published in Oxford Open Energy in October. It originated from a Freedom of Information request I submitted to the Greater London Authority last year.
It’s an open access journal, so here’s a copy. See the original article for references.
Citation
David Mytton, Masaō Ashtine, Scot Wheeler, David Wallom, Stretched grid? Managing data center energy demand and grid capacity, Oxford Open Energy, Volume 2, 2023. https://doi.org/10.1093/ooenergy/oiad014
Article
Data centers are a crucial component of information technology (IT) systems because they provide a cost-effective, reliable, and secure location for the computer equipment that powers most, if not all, the services we use daily—from video streaming and gaming to banking and email services. Although the definition of a data center can be broad—a facility containing IT equipment like servers and network devices—the trend is toward migration of workloads from older and smaller data centers toward large, “hyperscale” facilities operated by the biggest cloud providers such as Amazon, Google, and Microsoft. These facilities have at least 40 megawatts (MW) of power capacity, but some of the largest exceed 100 MW. Data centers also tend to be built in a small number of regions—68% of all data centers are in just 30 metro areas with Washington, DC/Northern Virginia, Shanghai, Beijing, Tokyo, and London accounting for 26% of the market.
In the UK, data centers have become highly concentrated in West London and the M4 corridor due to the proximity of fiber-optic cables that run undersea from North America making landfall in Cornwall and then travelling to London along that route. This siting is a major reason why the London boroughs of Ealing, Hounslow, and Hillingdon represent 90% of applications for grid connections from data centers over the last 2 years. Data centers are considered large energy users because of the high-density of IT equipment within. This load-heavy utilization is compounded by this concentration of data centers being within a small geographic area, placing pressure on available electricity distribution network operator (DNO) capacity when trying to balance varying stakeholder needs. The early signs of trouble appeared in a June 2022 article in The Economist, which revealed that battery storage systems and data centers were crowding out available DNO capacity. This quickly became mainstream news as the congestion was linked to multiyear delays to housing projects in West London, a particularly politicized issue.
A freedom of information (FOI) request revealed that the Greater London Authority had been informed of DNO capacity constraints by the city’s Housing and Land Infrastructure teams in March 2022. Due to this lack of capacity, Scottish and Southern Electricity Networks (SSEN), the regional DNO, provided local housing developers with a connection timeline as far out as 2030. The FOI response highlighted a high volume of requests for direct connections to the transmission network from data centers and battery installations, and SSEN further reported that capacity solutions were dependent on upgrades to the transmission network operated by National Grid Electricity Transmission not expected to be completed until 2029. The explanation by SSEN noted in the FOI response that data center connections were “unexpected” suggests a lack of communication and planning between data center owners, the regulators, the grid operators themselves, and local planning authorities, and it also highlights differences in construction timeframes. New data centers have historically taken up to 2 years to complete, but there are reports of faster buildouts completed in just 6–12 months. In Virginia, USA, where there are 275 data centers with a combined power capacity of 1908 MW in 2022, the local utility, Dominion, recently issued a warning that there was insufficient transmission capacity for new data centers. Globally, more than 300 hyperscale data centers are planned for construction, rapidly adding to the number already in operation and outpacing the ability of DNOs to add sufficient capacity.
The demand profile is clearly growing, so how should this be managed? There are limited options for grid operators or regional authorities such as cities to choose from, ranging from developing grid flexibility products to legislative restrictions. After a temporary ban in 2019, Amsterdam, a large data center hub, has now imposed annual power quotas for new data center construction with a maximum budget of 670 megavolt amperes (MVA) until 2030—these regional impositions will affect the construction of hyperscale data centers. Ireland is another major data center hub, and EirGrid’s capacity planning statement for 2021–30 indicated that data centers could account for 23% of all electricity demand in Ireland by 2030, though this projection has a wide range of uncertainty. Based on these forecasts, the Irish Commission for Regulation of Utilities (CRU) published a direction to the system operators outlining new, stricter criteria for connections. However, the use of speculative reservations was suggested as a reason for overprojecting capacity in Ireland—the Irish CRU reported that three-quarters of the capacity reserved was unused. Similarly, Hillingdon Council in London rebutted the idea that housing was being delayed due to a lack of electricity grid capacity, stating that capacity is often reserved but not used.
Managing capacity is clearly an area where policy can be improved. Data center buildings are usually constructed with excess capacity to allow growth, anticipating future customer demand, but beginning operations well under maximum capacity. Grid capacity is reserved for this maximum, but may be unused, blocking other users. Reservations currently operate on a first-come, first-served basis with no cost incentive to only reserve what is needed. Microsoft has suggested altering the pricing mechanism to introduce reservation pricing that scales based on usage and has an expiry date. As noted by the UK regulator, Ofgem, in 2020, DNOs could also adopt locational pricing for grid connections based on regional demand akin to models used to price electricity in other areas of the world. This could also flow back into pricing for data center customers—there are already regional differences in cloud pricing that reflect the varying costs of providing services in each location. If proximity to existing fiber cables is important, that should be reflected in the price, allowing the data center (and fiber network) operator to balance the cost of additional electricity grid capacity with building more fiber connections. If demand for connectivity is high enough, data center operators can contribute to the improvements necessary, as is the case with substation upgrades by data center operators in West London. Ofgem, having published its position in early 2022, is now reviewing options to incentivize the right behavior at different times and locations. Despite solutions being proposed by both regulators and industry, including Ofgem reiterating charging reform as a priority in November 2022, we have yet to see any definite progress with any of these approaches.
Aside from imposing capacity quotas, demand response is often discussed in the context of data centers because other large energy users, such as industrial refrigeration, can respond to requests to turndown events in return for lower energy or connection pricing agreements, or revenue within network flexibility markets. These demand response actions can both help balance energy supply and demand at a national level (increasingly important with greater variable renewable generation) or alleviate network power constraints at a local level. Enabled by the digitalization of the energy system, network operators globally are looking to flexibility to manage the network in a smarter way, either for short-term deferral of network upgrades (where there is uncertainty in future demand) or improving utilization of network infrastructure long-term by reducing the peaks that drive network reinforcement. In the UK, this is primarily endorsed by Ofgem and the UK Government’s smart systems and flexibility plan at a national level.
There are numerous pilot projects exploring flexibility at the distribution level such as EcoGrid 2.0 in Denmark, EnergyVille in Belgium and the UK-based TRANSITION, Electricity Flexibility and Forecasting Systems, and FUSION (TEF) collaboration and Project LEO (Local Energy Oxfordshire). Within the market trials run by Project LEO and TRANSITION, two categories of flexibility services were trialed: DSO Procured, where network operators pay for flexibility to avoid the network going beyond its firm capacity during known periods of high demand or following fault conditions, and DSO Enabled, which includes peer-to-peer trading of contracted network capacity that may allow local consortia to lower their combined network capacity and therefore connection costs. Another notable example of demand response flexibility is the recent world-first Demand Flexibility Service from National Grid ESO in the UK, which achieved more than 800 MWh in the first two events. However, despite several pilot projects, flexibility markets at the distribution level are still very much in their infancy.
Theoretically, data centers could participate in these types of flexibility programs either through temporary computing load reduction or relaxing the set points for the thermal conditions maintained through air-conditioning systems. However, adoption has so far been limited—few data centers have implemented these mechanisms because they clash with customer service-level agreements. The type of workloads that are suitable for location migration—those that are not latency sensitive—also tend to involve large volumes of data, e.g. training machine learning models, running analytics queries on data warehouses, or batch processing such as encoding videos. Migrating large amounts of data between data centers far enough away to be subject to a different grid mix would take lengthy periods because of network transfer times. Providing such capacity is also expensive, particularly if stored in cloud computing services, which have prohibitively high network egress fees between cloud regions. Further, the ability to shift IT workloads between locations requires designing the capability into IT software systems with redundancy that is expensive to implement and difficult to maintain, something only large organizations like Google have so far managed to achieve for specific workloads.
As discussed in Microsoft’s response to the Irish CRU directive, flexibility would not generally result in the data center “turning off.” Instead, data centers would go “off grid,” switching to on-site uninterruptible supplies such as diesel or gas generators. These are currently designed for emergency situations with batteries covering the data center load for a few seconds to a minute until generators are at speed and can take over. Aside from the risks associated with switching a live data center to a different source of energy, the carbon footprint of running fossil fuel-powered generators for extended periods of time runs counter to net zero goals. Some data centers are now being built with long-duration batteries designed to be charged whilst there is abundant clean energy, but this is not common due to the cost and experimental nature of such technologies. Microsoft has also just started a trial of providing grid services such as balancing via its on-site battery capacity.
At the heart of the issue is a mismatch of incentives. DNOs are under a reliability obligation to provide energy to existing customers, which means they have a conservative approach to network changes. In contrast, data center operators want to bring on new capacity quickly to address customer demand, and housing developers want to address the social and political imperative to build more. Considering the incentives involved is crucial to understanding the decisions made by the various stakeholders. Data center participation in demand response appears to be an intuitive solution to the overall problem, but only if the commercial incentives are ignored. Instead, data center operators are pushing for reform in other areas such as the pricing and management of capacity queues. Energy is a large part of the overall cost structure for data center operations, which impacts how decisions are made when choosing the location of new facilities, so including infrastructure and not just the cost at the meter would better align incentives. The design of specific demand response services and how they could be codeveloped with data center and grid operators would help link system requirements to appropriate pricing mechanisms.
Looking ahead, a collaborative effort among data center operators, regulators, grid operators, and local planning authorities will be imperative to harmonize the diverse interests and accelerate the necessary infrastructural and regulatory reforms. This means altering regulations and market design to represent the desires of the various parties and their relative priority more accurately. The lessons learned from these reforms will be important for the future as electricity demand shifts from small numbers of large-scale users (e.g. data centers and battery storage) to large numbers of smaller users caused by the increase in electric vehicles and heat pumps. The grid must evolve at all levels to deal with this change.