Changelog

Unreleased

Added

Include description of the egon-data workflow in our documentation #23
There's now a wrapper around subprocess.run in egon.data.subprocess.run. This wrapper catches errors better and displays better error messages than Python's built-in function. Use this wrapper wenn calling other programs in Airflow tasks.
You can now override the default database configuration using command line arguments. Look for the switches starting with --database in egon-data --help. See PR #159 for more details.
Docker will not be used if there is already a service listening on the HOST:PORT combination configured for the database.
You can now supply values for the command line arguments for egon-data using a configuration file. If the configuration file doesn't exist, it will be created by egon-data on it's first run. Note that the configuration file is read from and written to the directtory in which egon-data is started, so it's probably best to run egon-data in a dedicated directory. There's also the new function egon.data.config.settings which returns the current configuration settings. See PR #159 for more details.
You can now use tasks which are not part of a Dataset, i.e. which are unversioned, as dependencies of a dataset. See `PR #318`_ for more details.
You can now force the tasks of a Dataset to be always executed by giving the version of the Dataset a ".dev" suffix. See `PR #318`_ for more details.
OSM data import as done in open_ego #1 which was updated to the latest long-term data set of the 2021-01-01 in #223
Verwaltungsgebiete data import (vg250) more or less done as in open_ego #3
Zensus population data import #2
Zensus data import for households, apartments and buildings #91
DemandRegio data import for annual electricity demands #5
Download cleaned open-MaStR data from Zenodo #14
NEP 2021 input data import #45
Option for running workflow in test mode #112
Abstraction of hvmv and ehv substations #9
Filter zensus being inside Germany and assign population to municipalities #7
RE potential areas data import #124
Heat demand data import #101
Demographic change integration #47
Creation of voronoi polygons for hvmv and ehv substations #9
Add hydro and biomass power plants eGon2035 #127
Creation of the ehv/hv grid model with osmTGmod, see issue #4 and PR #164
Identification of medium-voltage grid districts #10
Distribute electrical demands of households to zensus cells #181
Distribute electrical demands of cts to zensus cells #210
Include industrial sites' download, import and merge #117
Integrate scenario table with parameters for each sector #177
The volume of the docker container for the PostgreSQL database is saved in the project directory under docker/database-data. The current user ($USER) is owner of the volume. Containers created prior to this change will fail when using the changed code. The container needs to be re-created. #228
Extract landuse areas from OSM #214
Integrate weather data and renewable feedin timeseries #19
Create and import district heating areas #162
Integrate electrical load time series for cts sector #109
Assign voltage level and bus_id to power plants #15
Integrate solar rooftop for etrago tables #255
Integrate gas bus and link tables #198
Integrate data bundle #272
Add household electricity demand time series, mapping of demand profiles to census cells and aggregated household electricity demand time series at MV grid district level #256
Integrate power-to-gas installation potential links #293
Integrate distribution of wind onshore and pv ground mounted generation #146
Integrate dynamic line rating potentials #72
Integrate gas voronoi polygons #308
Integrate supply strategies for individual and district heating #232
Integrate gas production #321
Integrate industrial time series creation #237
Merge electrical loads per bus and export to etrago tables #328
Insert industial gas demand #358
Integrate existing CHP and extdended CHP > 10MW_el #266
Add random seed to CLI parameters #351
Extend zensus by a combined table with all cells where there's either building, apartment or population data #359
Include allocation of pumped hydro units #332
Add example metadata for OSM, VG250 and Zensus VG250. Add metadata templates for licences, context and some helper functions. Extend docs on how to create metadata for tables. #139
Integrate DSM potentials for CTS and industry #259
Assign weather cell id to weather dependant power plants #330
Distribute wind offshore capacities #329
Add CH4 storages #405
Include allocation of conventional (non CHP) power plants #392
Fill egon-etrago-generators table #485
Include time-dependent coefficient of performance for heat pumps #532
Limit number of parallel processes per task #265
Include biomass CHP plants to eTraGo tables #498
Include Pypsa default values in table creation #544
Include PHS in eTraGo tables #333
Include feedin time series for wind offshore #531
Include carrier names in eTraGo table #551
Include hydrogen infrastructure for eGon2035 scenario #474
Include downloaded pypsa-eur-sec results #138
Create heat buses for eGon100RE scenario #582
Filter for DE in gas infrastructure deletion at beginning of respective tasks #567
Insert open cycle gas turbines into eTraGo tables #548
Preprocess buildings and amenities for LV grids #262
Assign household profiles to OSM buildings #435
Add link to meta creator to docs #599
Add extendable batteries and heat stores #566
Add efficiency, capital_cost and marginal_cost to gas related data in etrago tables #596
Add wind onshore farms for the eGon100RE scenario #690
The shared memory under "/dev/shm" is now shared between host and container. This was done because Docker has a rather tiny default for the size of "/dev/shm" which caused random problems. Guessing what size is correct is also not a good idea, so sharing between host and container seems like the best option. This restricts using egon-data with docker to Linux and MacOS, if the latter has "/dev/shm" but seems like the best course of action for now. Done via PR #703 and hopefully prevents issues #702 and #267 from ever occurring again.
Provide wrapper to catch DB unique violation #514
Add electric scenario parameters for eGon100RE #699
Introduce Sanity checks for eGon2035 #382
Add motorized individual travel #553
Allocating MaStR PV rooftop power plants to OSM and synthetic buildings. Desaggregating PV rooftop scenarios to mv grid districts and OSM and synthetic buildings. #684
Add mapping zensus - weather cells #845
Add pv rooftop plants per mv grid for eGon100RE #861
Integrated heavy duty transport FCEV #552
Assign CTS demands to buildings #671
Add sanity checks for residential electricity loads #902
Add sanity checks for cts loads #919
Add distribution of CHP plants for eGon100RE #851
Add mapping table for all used buildings #962
Add charging infrastructure for e-mobility #937
Add zipfile check #969
Add marginal costs for generators abroad and for carriers nuclear and coal #907
Add wind off shore power plants for eGon100RE #868
Write simBEV metadata to DB table PR #978
Add voltage level for electricity building loads #955
Add desaggregation of pv home batteries onto buildings #988
Desaggregation of DSM time series onto CTS consumers per bus id and individual indutry consumers. #1048
Add load areas #1014
Add new MaStR dataset #1051
Heat pump desaggregation to buildings PR #903
Add low flex scenario 'eGon2035_lowflex' #822
Add wrapped_partial to dynamise task generation #207

Changed

Adapt structure of the documentation to project specific requirements #20
Switch from Travis to GitHub actions for CI jobs #92
Rename columns to id and zensus_population_id in zensus tables #140
Revise docs CONTRIBUTING section and in particular PR guidelines #88 and #145
Drop support for Python3.6 #148
Improve selection of zensus data in test mode #151
Delete tables before re-creation and data insertation #166
Adjust residential heat demand in unpopulated zenus cells #167
Introduce mapping between VG250 municipalities and census cells #165
Delete tables if they exist before re-creation and data insertation #166
Add gdal to pre-requisites #185
Update task zensus-inside-germany #196
Update installation of demandregio's disaggregator #202
Update etrago tables #243 and #285
Migrate VG250 to datasets #283
Allow configuring the airflow port #281
Migrate mastr, mv_grid_districts and re_potential_areas to datasets #297
Migrate industrial sites to datasets #237
Rename etrago tables from e.g. egon_pf_hv_bus to egon_etrago bus etc. #334
Move functions used by multiple datasets #323
Migrate scenario tables to datasets #309
Migrate weather data and power plants to datasets #314
Create and fill table for CTS electricity demand per bus #326
Migrate osmTGmod to datasets #305
Filter osm landuse areas, rename industrial sites tables and update load curve function #378
Remove version columns from eTraGo tables and related code #384
Remove country column from scenario capacities table #391
Update version of zenodo download #397
Rename columns gid to id #169
Remove upper version limit of pandas #383
Use random seed from CLI parameters for CHP and society prognosis functions #351
Changed demand.egon_schmidt_industrial_sites - table and merged table (industrial_sites) #423
Replace 'gas' carrier with 'CH4' and 'H2' carriers #436
Adjust file path for industrial sites import #418
Rename columns subst_id to bus_id #335
Apply black and isort for all python scripts #463
Update deposit id for zenodo download #498
Add to etrago.setug.py the busmap table #484
Migrate dlr script to datasets #508
Migrate loadarea scripts to datasets #525
Migrate plot.py to dataset of district heating areas #527
Migrate substation scripts to datasets #304
Update deposit_id for zenodo download #540
Add household demand profiles to etrago table #381
Migrate zensus scripts to datasets #422
Add information on plz, city and federal state to data on mastr without chp #425
Assign residential heat demands to osm buildings #557
Add foreign gas buses and adjust cross bording pipelines #545
Integrate fuel and CO2 costs for eGon2035 to scenario parameters #549
Aggregate generators and stores for CH4 #629
Fill missing household data for populated cells #431
Fix RE potential areas outside of Germany by updating the dataset. Import files from data bundle. #592 #595
Add DC lines from Germany to Sweden and Denmark #611
H2 demand is met from the H2_grid buses. In Addtion, it can be met from the H2_saltcavern buses if a proximity criterion is fulfilled #620
Create H2 pipeline infrastructure for eGon100RE #638
Change refinement method for households types #651
H2 feed in links are changed to non extendable #653
Remove the '_fixed' suffix #628
Fill table demand.egon_demandregio_zensus_electricity after profile allocation #586
Change method of building assignment #663
Create new OSM residential building table #587
Move python-operators out of pipeline #644
Add annualized investment costs to eTraGo tables #672
Improve modelling of NG and biomethane production #678
Unify carrier names for both scenarios #575
Add automatic filtering of gas data: Pipelines of length zero and gas buses isolated of the grid are deleted. #590
Add gas data in neighbouring countries #727
Aggregate DSM components per substation #661
Aggregate NUTS3 industrial loads for CH4 and H2 #452
Update OSM dataset from 2021-02-02 to 2022-01-01 #486
Update deposit id to access v0.6 of the zenodo repository #627
Include electricity storages for eGon100RE scenario #581
Update deposit id to access v0.7 of the zenodo repository #736
Include simplified restrictions for H2 feed-in into CH4 grid #790
Update hh electricity profiles #735
Improve CH4 stores and productions aggregation by removing dedicated task #775
Add CH4 stores in Germany for eGon100RE #779
Assigment of H2 and CH4 capacitites for pipelines in eGon100RE #686
Update deposit id to access v0.8 of the zenodo repository #760
Add primary key to table openstreetmap.osm_ways_with_segments #787
Update pypsa-eur-sec fork and store national demand time series #402
Move and merge the two assign_gas_bus_id functions to a central place #797
Add coordinates to non AC buses abroad in eGon100RE #803
Integrate additional industrial electricity demands for eGon100RE #817
Set non extendable gas components from p-e-s as so for eGon100RE #877
Integrate new data bundle using zenodo sandbox #866
Add noflex scenario for motorized individual travel #821
Allocate PV home batteries to mv grid districts #749
Add sanity checks for motorized individual travel #820
Parallelize sanity checks #882
Insert crossboarding gas pipeline with Germany in eGon100RE #881
Rename noflex to lowflex scenario for motorized individual travel #921
Update creation of heat demand timeseries #857 #856
Overwrite retrofitted_CH4pipeline-to-H2pipeline_share with pes result #933
Adjust H2 industry profiles abroad for eGon2035 #940
Introduce carrier name 'others' #819
Add rural heat pumps per medium voltage grid district #987
Add eGon2021 scenario to demandregio dataset #1035
Update MaStR dataset #519
Add missing VOM costs for heat sector components #942
Desaggregate industry demands to OSM areas and industrial sites #1001
Add gas generator in Norway #1074
SQLAlchemy engine objects created via egon.data.db.engine are now cached on a per process basis, so only one engine is ever created for a single process. This fixes issue #799.
Insert rural heat per supply technology #1026
Change hgv data source to use database #1086

Bug Fixes

Some dependencies have their upper versions restricted now. This is mostly due to us not yet supporting Airflow 2.0 which means that it will no longer work with certain packages, but we also won't get and upper version limit for those from Airflow because version 1.X is unlikely to to get an update. So we had to make some implicit dependencies explicit in order to give them them upper version limits. Done via PR #692 in order to fix issues `#343`_, #556, #641 and #669.
Heat demand data import #157
Substation sequence #171
Adjust names of demandregios nuts3 regions according to nuts version 2016 #201
Delete zensus buildings, apartments and households in unpopulated cells #202
Fix input table of electrical-demands-zensus #217
Import heat demand raster files successively to fix import for dataset==Everything #204
Replace wrong table name in SQL function used in substation extraction #236
Fix osmtgmod for osm data from 2021 by updating substation in Garenfeld and set srid #241 #258
Adjust format of voltage levels in hvmv substation #248
Change order of osmtgmod tasks #253
Fix missing municipalities #279
Fix import of hydro power plants #270
Fix path to osm-file for osmtgmod_osm_import #258
Fix conflicting docker containers by setting a project name #289
Update task insert-nep-data for pandas version 1.3.0 #322
Fix versioning conflict with mv_grid_districts #340
Set current working directory as java's temp dir when executing osmosis #344
Fix border gas voronoi polygons which had no bus_id #362
Add dependency from WeatherData to Vg250 #387
Fix unnecessary columns in normal mode for inserting the gas production #390
Add xlrd and openpyxl to installation setup #400
Store files of OSM, zensus and VG250 in working dir #341
Remove hard-coded slashes in file paths to ensure Windows compatibility #398
Add missing dependency in pipeline.py #412
Add prefix egon to MV grid district tables #349
Bump MV grid district version no #432
Add curl to prerequisites in the docs #440
Replace NAN by 0 to avoid empty p_set column in DB #414
Exchange bus 0 and bus 1 in Power-to-H2 links #458
Fix missing cts demands for eGon2035 #511
Add data_bundle to industrial_sites task dependencies #468
Lift geopandas minimum requirement to 0.10.0 #504
Use inbuilt datetime package instead of pandas.datetime #516
Add missing 'sign' for CH4 and H2 loads #538
Delete only AC loads for eTraGo in electricity_demand_etrago #535
Filter target values by scenario name #570
Reduce number of timesteps of hh electricity demand profiles to 8760 #593
Fix assignemnt of heat demand profiles at German borders #585
Change source for H2 steel tank storage to Danish Energy Agency #605
Change carrier name from 'pv' to 'solar' in eTraGo_generators #617
Assign "carrier" to transmission lines with no value in grid.egon_etrago_line #625
Fix deleting from eTraGo tables #613
Fix positions of the foreign gas buses #618
Create and fill transfer_busses table in substation-dataset #610
H2 steel tanks are removed again from saltcavern storage #621
Timeseries not deleted from grid.etrago_generator_timeseries #645
Fix function to get scaled hh profiles #674
Change order of pypsa-eur-sec and scenario-capacities #589
Fix gas storages capacities #676
Distribute rural heat supply to residetntial and service demands #679
Fix time series creation for pv rooftop #688
Fix extraction of buildings without amenities #693
Assign DLR capacities to every transmission line #683
Fix solar ground mounted total installed capacity #695
Fix twisted number error residential demand #704
Fix industrial H2 and CH4 demand for eGon100RE scenario #687
Clean up "pipeline.py" #562
Assign timeseries data to crossborder generators ego2035 #724
Add missing dataset dependencies in "pipeline.py" #725
Fix assignemnt of impedances (x) to etrago tables #710
Fix country_code attribution of two gas buses #744
Fix voronoi assignemnt for enclaves #734
Set lengths of non-pipeline links to 0 #741
Change table name from boundaries.saltstructures_inspee to boundaries.inspee_saltstructures #746
Add missing marginal costs for conventional generators in Germany #722
Fix carrier name for solar ground mounted in scenario parameters #752
Create rural_heat buses only for mv grid districts with heat load #708
Solve problem while creating generators series data egon2035 #758
Correct wrong carrier name when assigning marginal costs #766
Use db.next_etrago_id in dsm and pv_rooftop dataset #748
Add missing dependency to heat_etrago #771
Fix country code of gas pipeline DE-AT #813
Fix distribution of resistive heaters in district heating grids #783
Fix missing reservoir and run_of_river power plants in eTraGo tables, Modify fill_etrago_gen to also group generators from eGon100RE, Use db.next_etrago_id in fill_etrago_gen #798 #776
Fix model load timeseries in motorized individual travel #830
Fix gas costs #847
Add imports that have been wrongly deleted #849
Fix final demand of heat demand timeseries #781
Add extendable batteries only to buses at substations #852
Move class definition for grid.egon_gas_voronoi out of etrago_setup #888
Temporarily set upper version limit for pandas #829
Change industrial gas load modelling #871
Delete eMob MIT data from eTraGo tables on init #878
Fix model id issues in DSM potentials for CTS and industry #901
Drop isolated buses and tranformers in eHV grid #874
Model gas turbines always as links #914
Drop era5 weather cell table using cascade #909
Remove drop of p_set and q_set for loads without timeserie #971
Delete gas bus with wrong country code #958
Overwrite capacities for conventional power plants with data from nep list #403
Make gas grid links bidirectional #1021
Correct gas technology costs for eGon100RE #984
Adjust p_nom and marginal cost for OCGT in eGon2035 #863
Mismatch of building bus_ids from cts_heat_demand_building_share and mapping table #989
Fix zensus weather cells mapping #1031
Fix solar rooftop in test mode #1055
Add missing filter for scenario name in chp expansion #1015
Fix installed capacity per individual heat pump #1058
Add missing gas turbines abroad #1079
Fix gas generators abroad (marginal cost and e_nom_max) #1075
Fix gas pipelines isolated of the German grid #1081
Fix aggregation of DSM-components #1069
Fix URL of TYNDP scenario dataset
Automatically generated tasks now get unique task_ids. Fixes issue #985 via PR #986.
Fix faulty DSM time series #1088
Set upper limit on commissioning date for units from MaStR dataset #1098
Fix conversion factor for CH4 loads abroad in eGon2035 #1104
Fix URLs of MaStR datasets
Fix CRS in ERA5 transformation #179
Fix depricated python-operator import #183
Fix incorrect YAML serialization of the --scenarios CLI parameter that caused broken DAGs in the pipeline. #343