Consumption time series manipulation¶

Table of Contents¶

  • 1. Thermal sensitivity of consumption
    • 1.1. Definition
    • 1.2. Functions to estimate thermosensitivity and decompose consumption
    • 1.3. Functions for modification of temperature and thermosensitivity of consumption
    • 1.4. Some open questions
  • 2. Modeling with average profiles
    • 2.1 Sector by sector version simple [Tertiaire,résidentiel,Indus,Autre]x[ChauffagexAutre]
    • 2.2 Sector by sector version [Tertiaire,résidentiel,Indus,Autre]x[ChauffagexECSxCuissonx...]
    • 2.3 Electric Vehicle consumption
  • 3. Modeling with a more physical approach. Cas of Heat-pump/hybrid heat-pump model

This notebook gives a detailed explanation of the code contained in Consumption_TS_manipulation_examples.py.

In this notebook, you will manipulate temperature time series, together with models of the thermal sensitivity of consumption.

In the end you will learn how to

  • work with thermal sensitivity :
    • estimate thermal sensitivity given time series of consumption and temperature
    • decompose a consumption into two parts. The first one will not be correlated with temperature and the other one will depend linearly on cold temperatures
    • remove the effect of temperature from year "x" in a consumption time series and apply the temperature from year "y".
    • use a thermal sensitive consumption model to model electric car consumption
  • work with profiles to decompose consumptions :
    • EV-profiles/models, Water-heater profiles, decomposed thermal sensitive and non thermal sensitive
    • sector by sector profiles, total consumption decomposition
  • work with more advanced consumption models :
    • heat pump, hybrid heat pump ...

Make sure this is where the data folder is (the one in Basic_France_models/Consumption/):

In [1]:
InputFolder='Data/'

Add root directory to path, and load the function hide_toggle that can be used in cells to hide content

In [2]:
import os
import sys
if os.path.basename(os.getcwd())=='Consumption':
    sys.path.append('../../../')
from functions.f_notebook import hide_toggle

Importation of module – please be sure that you are running the notebook under the energy-alternatives environment

In [3]:
#region importation of modules

import numpy as np
import seaborn as sns
import pandas as pd
import csv
import datetime
import copy
import plotly.graph_objects as go
import matplotlib.pyplot as plt
from sklearn import linear_model
from functions.f_consumptionModels import * #Il faut préciser le chemin où vous avez sauvegardé les données csv
from functions.f_graphicalTools import * #Il faut préciser le chemin où vous avez sauvegardé les données csv
from functions.f_heat_pump import *
hide_toggle() ## this adds a button that allows you to hide content
#endregion
Out[3]:
Toggle show/hide

1. Thermal sensitivity of consumption ¶

1.1. Definition ¶

As an introduction, french speakers can read my post (contribution for translation of this site's posts are welcome) on the subject.

You can observe how consumption depends upon temperature for year 2012. Below a threshold temperature, here around 15 degres, the relation between consumption and temperature is linear. The coefficient of the linear regression is, by definition, what we call the thermal sensitivity.

In [4]:
#region  Load and visualize consumption
ConsoTempe_df=pd.read_csv(InputFolder+'ConsumptionTemperature_1996TO2019_FR.csv',parse_dates=['Date']).set_index(["Date"])
year = 2012
ConsoTempeYear_df=ConsoTempe_df.loc[str(year)]
hour = 19
TemperatureThreshold = 15
plt.plot(ConsoTempeYear_df['Temperature'],ConsoTempeYear_df['Consumption']/1000, '.', color='black');
plt.show()
#endregion

1.2. Functions to estimate thermosensitivity and decompose consumption ¶

Formally, the preceding linear regression is given by the following formula:

$$ C_t = C^{NT}_t+\rho_{h(t)}(T_t-T_0)_+$$

where $T_0$ is the threshold temperature, $(x)_+$ is $x$ for any non negative $x$ and zero otherwise, $\rho_{h(t)}$ is the thermal sensitivity for hour $h$ (if it depends upon the hour of the day) $C^{NT}_t$ is the part of the consumption that is not sensitive to the temperature. It is obtained as the residuals of the linear regression for temperatures lower than $T_0$ and is the consumption itself otherwise.

In [5]:
#select dates to do the linear regression
indexHeatingHour = (ConsoTempeYear_df['Temperature'] <= TemperatureThreshold) &\
                    (ConsoTempeYear_df.index.to_series().dt.hour == hour)
ConsoHeatingHour= ConsoTempeYear_df[indexHeatingHour]
lr=linear_model.LinearRegression().fit(ConsoHeatingHour[['Temperature']],
                                       ConsoHeatingHour['Consumption'])
lr.coef_[0]
Out[5]:
-2411.201355839509

In module EnergyAlternativesPlanning.f_consumptionModels, we have implemented a function to estimate the different elements in the preceding equation, given consumption, temperature and other parameters (such as $T_0$). Here is an example of how you can use it, but you can have a look at the function Decomposeconso.

In [6]:
#region  Thermal sensitivity estimation, consumption decomposition and visualisation

#Generic function Thermal sensitivity estimation
(ConsoTempeYear_decomposed_df,Thermosensibilite)=Decomposeconso(ConsoTempeYear_df,TemperatureThreshold=TemperatureThreshold)
#ConsoTempeYear_decomposed_df=ConsoTempeYear_decomposed_df.rename(columns={'NTS_C':'Conso non thermosensible', "TS_C": 'conso thermosensible'})
fig=MyStackedPlotly(y_df=ConsoTempeYear_decomposed_df[["NTS_C","TS_C"]],
                    Names=['Conso non thermosensible','conso thermosensible'])
fig=fig.update_layout(title_text="Consommation (MWh)", xaxis_title="Date")
#plotly.offline.plot(fig, filename='file.html') ## offline
fig.show()
#endregion

1.3. Functions for modification of temperature and thermosensitivity of consumption ¶

You can do other interesting things. For example, you can redecompose the electric consumption of the year X to thermosensitive and no-thermosensitive parts from the thermosensitivity of the year X and the temperatures of the year Y. It's very useful to compare the years.

In [7]:
#region Thermal sensitivity model to change meteo
## change meteo year
## example for year 2012
newyear=2012
NewConsoTempeYear_df = ConsoTempe_df.loc[str(newyear)]
(ConsoTempeYear_decomposed_df,Thermosensibilite)=Decomposeconso(NewConsoTempeYear_df,TemperatureThreshold=TemperatureThreshold)

NewConsoTempeYear_decomposed_df=Recompose(ConsoTempeYear_decomposed_df,Thermosensibilite,
                                          Newdata_df=NewConsoTempeYear_df,
                                          TemperatureThreshold=TemperatureThreshold)
### loop over years
fig = go.Figure()
TMP=ConsoTempeYear_decomposed_df.copy()
TMP = TMP.reset_index().drop(columns="Date").assign(Date=range(1, len(TMP) + 1)).set_index(["Date"])
fig = fig.add_trace(
    go.Scatter(x=TMP.index,y=ConsoTempeYear_decomposed_df['Consumption'],line=dict(color="#000000"),name="original"))
for newyear in range(2000,2012):
    NewConsoTempeYear_df = ConsoTempe_df.loc[str(newyear)]
    ConsoSepareeNew_df=Recompose(ConsoTempeYear_decomposed_df,Thermosensibilite,
                                 Newdata_df=NewConsoTempeYear_df,
                                 TemperatureThreshold=TemperatureThreshold)
    ConsoSepareeNew_df = ConsoSepareeNew_df.reset_index().drop(columns="Date").assign(
        Date=range(1, len(ConsoSepareeNew_df) + 1)).set_index(["Date"])

    fig.add_trace(go.Scatter(x=ConsoSepareeNew_df.index,
                             y=ConsoSepareeNew_df['Consumption'],
                             line=dict(color="#9CA2A8",width=1),
                             name=newyear))

fig.show()
#endregion

In addition, with the same function Recompose to that you can also redecompose the electric consumption of the year X to thermosensitive and no-thermosensitive parts from a new table of thermosensitivity.

In [8]:
#region Thermal sensitivity model to change thermal sensitivity
## change thermal sensitivity
NewThermosensibilite={}
for key in Thermosensibilite:    NewThermosensibilite[key]=1/3 * Thermosensibilite[key]
NewConsoTempeYear_decomposed_df=Recompose(ConsoTempeYear_decomposed_df,NewThermosensibilite,
                                          TemperatureThreshold=TemperatureThreshold)
fig = go.Figure()
fig.add_trace(go.Scatter(x=ConsoTempeYear_decomposed_df.index,
                         y=ConsoTempeYear_decomposed_df['Consumption'],
                         line=dict(color="#000000"),name="original"))
fig.add_trace(go.Scatter(x=NewConsoTempeYear_decomposed_df.index,
                             y=NewConsoTempeYear_decomposed_df['Consumption'],
                             line=dict(color="#9CA2A8",width=1),
                             name=newyear))
#plotly.offline.plot(fig, filename='file.html') ## offline
fig.show()
#endregion

1.4. Further questions ¶

Now, you know the basics about thermosensitivity, here are some questions that you can explore now or later :

  • Q1 - What usage causes thermal sensitivity ? is it only heating ? what about electric network losses, cooking, water heater, electric vehicle, ... How can you model that ?
  • Q2 - What is the relation between building efficiency (now and in the future) as studied in this notebook and the thermal sensitivity defined here.
  • Q3 - How heat pump, and hybrid heat-pump will change de thermal sensitivity
  • Q4 - For high temperature, what are the reason for thermal sensitivity to be stronger ? smaller ?
  • Q5 - how would you define the "national temperature"
  • Q6 - want to contribute ? have a look at eco2mix territorial data (for french regions, or big city), how these data can be used to improve knowledge on the different questions here ?

2. Modeling with average profiles ¶

Average profiles are profiles of consumption defined for a tipical day of several week days. It can depend on the day of the week, or just be different depending on weekday/saturday/sunday. Several profiles can be build for different seasons. In examples below, they are used for

  • a top-down model where the national total hourly consumption is split into different sector (and usages) according to sector/usage-profiles

  • (a kind of) bottom-up modelin of electric vehicle and water heater.

2.1 Sector by sector version simple [Tertiaire,résidentiel,Indus,Autre]x[ChauffagexAutre] ¶

In [9]:
#region consumption decomposition [Tertiaire,résidentiel,Indus,Autre]x[ChauffagexAutre]
year = 2012
TemperatureThreshold = 15
ConsoTempe_df=pd.read_csv(InputFolder+'ConsumptionTemperature_1996TO2019_FR.csv',parse_dates=['Date']).\
    set_index(["Date"]).loc[str(year)]
ConsoTempe_df=ConsoTempe_df[~ConsoTempe_df.index.duplicated(keep='first')]
(ConsoTempeYear_decomposed_df,Thermosensibilite)=Decomposeconso(ConsoTempe_df,TemperatureThreshold=TemperatureThreshold)

Profile_df_sans_chauffage=pd.read_csv(InputFolder+"ConsumptionDetailedProfiles.csv").\
    rename(columns={'heures':'Heure',"WeekDay":"Jour"}).\
    replace({"Jour" :{"Sat": "Samedi" , "Week":"Semaine"  , "Sun": "Dimanche"}}). \
    query('UsagesGroupe != "Chauffage"'). \
    set_index(["Mois", "Heure",'Nature', 'type', 'UsagesGroupe', 'UsageDetail', "Jour"]).\
    groupby(["Mois","Jour","Heure","type"]).sum().\
    merge(add_day_month_hour(df=ConsoTempe_df,semaine_simplifie=True,French=True,to_index=True),
          how="outer",left_index=True,right_index=True).reset_index().set_index("Date")[["type","Conso"]]. \
    pivot_table(index="Date", columns=["type"], values='Conso')

Profile_df_sans_chauffage=Profile_df_sans_chauffage.loc[:,Profile_df_sans_chauffage.sum(axis=0)>0]
Profile_df_n=Profile_df_sans_chauffage.div(Profile_df_sans_chauffage.sum(axis=1), axis=0) ### normalisation par 1 et multiplication
for col in Profile_df_sans_chauffage.columns:
    Profile_df_sans_chauffage[col]=Profile_df_n[col]*ConsoTempeYear_decomposed_df["NTS_C"]

Profile_df_seulement_chauffage=pd.read_csv(InputFolder+"ConsumptionDetailedProfiles.csv").\
    rename(columns={'heures':'Heure',"WeekDay":"Jour"}).\
    replace({"Jour" :{"Sat": "Samedi" , "Week":"Semaine"  , "Sun": "Dimanche"}}). \
    query('UsagesGroupe == "Chauffage"'). \
    set_index(["Mois", "Heure",'Nature', 'type', 'UsagesGroupe', 'UsageDetail', "Jour"]).\
    groupby(["Mois","Jour","Heure","type"]).sum().\
    merge(add_day_month_hour(df=ConsoTempe_df,semaine_simplifie=True,French=True,to_index=True),
          how="outer",left_index=True,right_index=True).reset_index().set_index("Date")[["type","Conso"]]. \
    pivot_table(index="Date", columns=["type"], values='Conso')

Profile_df_seulement_chauffage=Profile_df_seulement_chauffage.loc[:,Profile_df_seulement_chauffage.sum(axis=0)>0]
Profile_df_n=Profile_df_seulement_chauffage.div(Profile_df_seulement_chauffage.sum(axis=1), axis=0) ### normalisation par 1 et multiplication
for col in Profile_df_seulement_chauffage.columns:
    Profile_df_seulement_chauffage[col]=Profile_df_n[col]*ConsoTempeYear_decomposed_df["TS_C"]

Profile_df_seulement_chauffage.columns=[(col,"chauffage") for col in Profile_df_seulement_chauffage.columns]
Profile_df_sans_chauffage.columns=[(col,"autre") for col in Profile_df_sans_chauffage.columns]
Profile_df=pd.concat([Profile_df_seulement_chauffage,Profile_df_sans_chauffage],axis=1)
Profile_df.columns=pd.MultiIndex.from_tuples(Profile_df.columns, names=('type', 'usage'))
fig = MyStackedPlotly(y_df=Profile_df)
#plotly.offline.plot(fig, filename='file.html') ## offline
fig.show()
#endregion

Look at the energy consumption per usage/sector in TWh :

In [10]:
Profile_df.sum(axis=0)/10**6
Out[10]:
Ind_chauffage              9.137457
Residentiel_chauffage     54.726530
Tertiaire_chauffage       21.938555
Autre_autre                7.513457
Ind_autre                 53.927463
Residentiel_autre        159.301062
Tertiaire_autre          180.087520
dtype: float64

2.3 Sector by sector version [Tertiaire,résidentiel,Indus,Autre]x[ChauffagexECSxCuissonx...] ¶

In [11]:
#region consumption decomposition [Tertiaire,résidentiel,Indus,Autre]x[ChauffagexECSxCuissonx...]
year = 2012
ConsoTempe_df=pd.read_csv(InputFolder+'ConsumptionTemperature_1996TO2019_FR.csv',parse_dates=['Date']).\
    set_index(["Date"]).loc[str(year)]
ConsoTempe_df=ConsoTempe_df[~ConsoTempe_df.index.duplicated(keep='first')]
(ConsoTempeYear_decomposed_df,Thermosensibilite)=Decomposeconso(ConsoTempe_df,TemperatureThreshold=TemperatureThreshold)

UsagesGroupe_simplified_dict={'Autres': "Specifique et autre",  'Congelateur':"Specifique et autre",
       'Eclairage': "Specifique et autre" ,
       'LaveLinge':"Specifique et autre", 'Lavevaisselle':"Specifique et autre",
       'Ordis': "Specifique et autre",
       'Refrigirateur' :"Specifique et autre" , 'SecheLinge' :"Specifique et autre",
        'TVAutreElectroMen' : "Specifique et autre"}

Profile_df_sans_chauffage=pd.read_csv(InputFolder+"ConsumptionDetailedProfiles.csv").\
    rename(columns={'heures':'Heure',"WeekDay":"Jour"}).\
    replace({"Jour" :{"Sat": "Samedi" , "Week":"Semaine"  , "Sun": "Dimanche"},
             'UsagesGroupe':UsagesGroupe_simplified_dict}). \
    query('UsagesGroupe != "Chauffage"'). \
    set_index(["Mois", "Heure",'Nature', 'type','UsagesGroupe', 'UsageDetail', "Jour"]).\
    groupby(["Mois","Jour","Heure",'type','UsagesGroupe']).sum().\
    merge(add_day_month_hour(df=ConsoTempe_df,semaine_simplifie=True,French=True,to_index=True),
          how="outer",left_index=True,right_index=True).reset_index().set_index("Date")[['type','UsagesGroupe',"Conso"]]. \
    pivot_table(index="Date", columns=['type','UsagesGroupe'], values='Conso')

Profile_df_sans_chauffage=Profile_df_sans_chauffage.loc[:,Profile_df_sans_chauffage.sum(axis=0)>0]
Profile_df_n=Profile_df_sans_chauffage.div(Profile_df_sans_chauffage.sum(axis=1), axis=0) ### normalisation par 1 et multiplication
for col in Profile_df_sans_chauffage.columns:
    Profile_df_sans_chauffage[col]=Profile_df_n[col]*ConsoTempeYear_decomposed_df["NTS_C"]


Profile_df_seulement_chauffage=pd.read_csv(InputFolder+"ConsumptionDetailedProfiles.csv").\
    rename(columns={'heures':'Heure',"WeekDay":"Jour"}).\
    replace({"Jour" :{"Sat": "Samedi" , "Week":"Semaine"  , "Sun": "Dimanche"},
             'UsagesGroupe':UsagesGroupe_simplified_dict}). \
    query('UsagesGroupe == "Chauffage"'). \
    set_index(["Mois", "Heure",'Nature', 'type','UsagesGroupe', 'UsageDetail', "Jour"]).\
    groupby(["Mois","Jour","Heure",'type','UsagesGroupe']).sum().\
    merge(add_day_month_hour(df=ConsoTempe_df,semaine_simplifie=True,French=True,to_index=True),
          how="outer",left_index=True,right_index=True).reset_index().set_index("Date")[['type','UsagesGroupe',"Conso"]]. \
    pivot_table(index="Date", columns=['type','UsagesGroupe'], values='Conso')

Profile_df_seulement_chauffage=Profile_df_seulement_chauffage.loc[:,Profile_df_seulement_chauffage.sum(axis=0)>0]
Profile_df_n=Profile_df_seulement_chauffage.div(Profile_df_seulement_chauffage.sum(axis=1), axis=0) ### normalisation par 1 et multiplication
for col in Profile_df_seulement_chauffage.columns:
    Profile_df_seulement_chauffage[col]=Profile_df_n[col]*ConsoTempeYear_decomposed_df["TS_C"]

Profile_df=pd.concat([Profile_df_seulement_chauffage,Profile_df_sans_chauffage],axis=1)
fig = MyStackedPlotly(y_df=Profile_df)
#plotly.offline.plot(fig, filename='file.html') ## offline
fig.show()

2.3 Electric Vehicle consumption ¶

In [12]:
ConsoTempe_df=pd.read_csv(InputFolder+'ConsumptionTemperature_1996TO2019_FR.csv',parse_dates=['Date']).set_index(["Date"]).loc[str(year)]
VEProfile_df=pd.read_csv(InputFolder+'EVModel.csv', sep=';')
VEProfile_df.head()
Out[12]:
Saison Jour Heure Puissance.MW.par.million
0 Hiver 2 0 600
1 Hiver 2 1 500
2 Hiver 2 2 350
3 Hiver 2 3 220
4 Hiver 2 4 160

Let us now transform this profile into a time series. Here the function Profile2Consumption generates a basic consumption based on the "summer" profile, and a thermal sensitive part. The coefficients of the thermal sensitive part are estimated with the Temperature data and the difference between the summer profile and the winter profile. Have a look at the code.

In [13]:
year=2012
EV_Consumption_df=Profile2Consumption(Profile_df=VEProfile_df,Temperature_df = ConsoTempe_df.loc[str(year)][['Temperature']])
fig=MyStackedPlotly(y_df=EV_Consumption_df[["NTS_C","TS_C"]],
                    Names=['Conso VE non thermosensible','conso VE thermosensible'])
fig=fig.update_layout(title_text="Consommation (MWh)", xaxis_title="Date")
#plotly.offline.plot(fig, filename='file.html') ## offline
fig.show()

Now the same with Water heater

In [14]:
#même chose avec l'ECS
ECS_Profile_df=pd.read_csv(InputFolder+'Profil_ECS_RTE.csv',sep=';',decimal=',',encoding='utf-8').\
    melt(id_vars=["Jour","Heure"],value_vars=["ECS en juin","ECS en janvier"],value_name="Puissance.MW",var_name="Saison").\
    replace({"Saison":{"ECS en juin":"Ete","ECS en janvier":"Hiver"},
             "Jour":{"Lundi":1,"Mardi":2,"Mercredi":3,"Jeudi":4,"Vendredi":5,"Samedi":6,"Dimanche":7}})
ECS_Profile_df
Out[14]:
Jour Heure Saison Puissance.MW
0 1 0 Ete 5736
1 1 1 Ete 5250
2 1 2 Ete 3702
3 1 3 Ete 2129
4 1 4 Ete 1633
... ... ... ... ...
331 7 19 Hiver 1670
332 7 20 Hiver 1858
333 7 21 Hiver 2635
334 7 22 Hiver 7835
335 7 23 Hiver 10369

336 rows × 4 columns

In [15]:
ECS_Consumption_df=Profile2Consumption(Profile_df=ECS_Profile_df,Temperature_df = ConsoTempe_df.loc[str(year)][['Temperature']],VarName="Puissance.MW")
fig=MyStackedPlotly(y_df=ECS_Consumption_df[["NTS_C","TS_C"]],
                    Names=['Conso ECS non thermosensible','conso ECS thermosensible'])
fig=fig.update_layout(title_text="Consommation (MWh)", xaxis_title="Date")
#plotly.offline.plot(fig, filename='file.html') ## offline
fig.show()

3. Heat-pump/hybrid heat-pump model ¶

Here, a heat pump model is proposed. See more details in

In [16]:
#region heat pump
ConsoTempe_df=pd.read_csv(InputFolder+'ConsumptionTemperature_1996TO2019_FR.csv',parse_dates=['Date']).set_index(["Date"]).\
                    rename(columns={'Temperature' :'temp'})[["temp"]]
year=2018
Simulation_PAC_input_parameter = {
    "System": "A/A HP", "Technology": "Inverter", "Mode": "Bivalent", "Emitters": "Fan coil unit",
    "N_stages": np.nan, "Power_ratio": 3.0, "PLF_biv": 1.4, "Ce": 0.7, "Lifetime": 17, "Temperature_limit": -10,
    "Share_Power": 0.5, "regulation": "Y", "T_start" : 15, "T_target" : 18   }

SCOP=estim_SCOP(ConsoTempe_df, Simulation_PAC_input_parameter,year=year)
MyConsoData =SCOP["meteo_data_heating_period"][["P_calo",'P_app',"P_elec"]]
index_year =ConsoTempe_df.loc[str(year)].index
MyConsoData_filled=pd.DataFrame([0]*len(index_year),index=index_year).\
    merge(MyConsoData,how="outer",left_index=True,right_index=True).drop(columns=0).fillna(0)
fig=MyStackedPlotly(y_df=MyConsoData_filled[["P_calo",'P_app']])
fig.add_trace(go.Scatter(x=MyConsoData_filled.index,
                         y=MyConsoData_filled["P_elec"], name="Puissance PAC",
                         line=dict(color='red', width=0.4)))
fig=fig.update_layout(title_text="Conso (en Delta°C)", xaxis_title="heures de l'année")
#plotly.offline.plot(fig, filename='file.html') ## offline
fig.show()
#endregion
In [ ]: