Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create metadata for existing datasets #450

Open
nesnoj opened this issue Sep 22, 2021 · 9 comments · May be fixed by #1122
Open

Create metadata for existing datasets #450

nesnoj opened this issue Sep 22, 2021 · 9 comments · May be fixed by #1122
Assignees
Labels
🏗️ integration Integrating a data processing step 🙏 help wanted Extra attention is needed

Comments

@nesnoj
Copy link
Member

nesnoj commented Sep 22, 2021

Now there're a howto and examples on metadata #176. Now it's up to all of us:

Everyone, please

  1. Check the DB table list below and
  • Tick the column Metadata needed? if you think this is a table which needs further explanation (at least all tables we're up to publish on the OEP List tables for OEP release  #367 do need metadata)
  • Add your nick for datasets you're responsible for
  • Tick Done! if you're done, even it has not been merged to dev yet
  1. Create metadata for your tables and make sure they comply with the OEP 1.4.1 standard. Read the instructions how to do so, see finished datasets below for examples.

Notes

  • There're some helpers (e.g. a bunch of standard licenses), see instructions. See examples on how to use them.
  • You do not have to set the fields id and publicationDate as they will be set in a final metadata task Verify, enrich and export metadata #451. But you have to set them to make OMI 👵 happy, example.
  • A new task Verify, enrich and export metadata #451 will be created which will be executed at the end of the pipeline that checks all metadata against the OEP 1.4.1 standard via OMI tool (regardless of this final check, please make sure your metadata string adheres to the standard).

Examples: There're OMI-approved metadata for datasets of 3 different tasks:

If you have further questions or remarks, please drop me a line...

List of DB tables

table_schema table_name Metadata needed? Responsible person Done!
boundaries egon_map_mvgriddistrict_vg250 YES @ClaraBuettner @IlkaCu
boundaries egon_map_zensus_buildings_filtered YES (Really?)
boundaries egon_map_zensus_grid_districts YES @ClaraBuettner @IlkaCu
boundaries egon_map_zensus_vg250 YES @ClaraBuettner @IlkaCu
boundaries vg250_gem YES @nesnoj X
boundaries vg250_gem_clean YES @nesnoj
boundaries vg250_gem_population YES @nesnoj X
boundaries vg250_krs YES @nesnoj X
boundaries vg250_lan YES @nesnoj X
boundaries vg250_rbz YES @nesnoj X
boundaries vg250_sta YES @nesnoj X
boundaries vg250_vwg YES @nesnoj X
boundaries bkg_vg250_statistics_view
boundaries egon_map_zensus_climate_zones YES
boundaries inspee_saltstructures
boundaries egon_map_zensus_mvgd_buildings YES @nailend
boundaries egon_map_zensus_vg250
boundaries egon_map_zensus_buildings_filtered_all
boundaries egon_map_zensus_buildings_residential
boundaries egon_map_zensus_weather_cell YES
demand egon_demandregio_cts_ind NO
demand egon_demandregio_hh NO
demand egon_demandregio_osm_ind_electricity NO
demand egon_demandregio_sites_ind_electricity NO
demand egon_demandregio_timeseries_cts_ind NO
demand egon_demandregio_wz NO
demand egon_demandregio_zensus_electricity YES @ClaraBuettner
demand egon_district_heating_areas YES @ClaraBuettner
demand egon_etrago_electricity_cts YES
demand egon_etrago_heat_cts YES
demand egon_etrago_electricity_households NO
demand egon_hotmaps_industrial_sites NO
demand egon_household_electricity_profile_in_census_cell YES @nesnoj
demand egon_industrial_sites YES @IlkaCu
demand egon_map_zensus_district_heating_areas YES @ClaraBuettner
demand egon_osm_ind_load_curves NO
demand egon_peta_heat YES @ClaraBuettner
demand egon_schmidt_industrial_sites NO
demand egon_seenergies_industrial_sites NO
demand egon_sites_ind_load_curves NO
demand iee_household_load_profiles (check PR523) YES @nesnoj
demand egon_cts_heat_demand_building_share YES
demand egon_cts_electricity_demand_building_share YES
demand egon_building_electricity_peak_loads YES
demand egon_loadarea YES
demand egon_timeseries_district_heating YES
demand egon_building_heat_peak_loads
demand egon_household_electricity_profile_of_buildings YES
demand egon_osm_ind_load_curves_individual YES
demand egon_sites_ind_load_curves
demand egon_sites_ind_load_curves_individual YES
demand egon_etrago_electricity_cts_dsm_timeseries
demand egon_osm_ind_load_curves_individual_dsm_timeseries
demand egon_heavy_duty_transport_voronoi
demand egon_daily_heat_demand_per_climate_zone YES
demand egon_hp_capacity_buildings YES
demand egon_ev_count_registration_district
demand egon_ev_count_municipality
demand egon_ev_count_mv_grid_district
demand egon_ev_pool
demand egon_ev_trip YES
demand egon_ev_mv_grid_district YES
demand egon_ev_metadata YES
demand egon_heat_idp_pool YES
demand egon_heat_timeseries_selected_profiles YES
grid egon_ehv_substation YES @IlkaCu
grid egon_ehv_substation_voronoi NO
grid egon_etrago_bus YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_bus_timeseries YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_carrier YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_generator YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_generator_timeseries YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_line YES @ClaraBuettner
grid egon_etrago_line_timeseries YES @ClaraBuettner
grid egon_etrago_link YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_link_timeseries YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_load YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_load_timeseries YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_storage YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_storage_timeseries YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_store YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_store_timeseries YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_temp_resolution YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_transformer YES @ClaraBuettner @AmeliaNadal
grid egon_etrago_transformer_timeseries YES @ClaraBuettner @AmeliaNadal
grid egon_voronoi_ch4 NO
grid egon_voronoi_h2 NO
grid egon_hvmv_substation YES @IlkaCu
grid egon_hvmv_substation_voronoi NO
grid egon_mv_grid_district (check PR523) YES @nesnoj
grid osm_point
grid osm_polygon
grid osm_roads
grid destatis_zensus_population_per_ha_inside_germany
grid egon_destatis_zensus_building_per_ha
grid vg250_gem_population
grid egon_destatis_zensus_apartment_per_ha
grid egon_building_electricity_peak_loads
grid egon_destatis_zensus_apartment_building_population_per_ha
grid egon_destatis_zensus_household_per_ha
grid egon_population_prognosis
grid egon_household_prognosis
grid egon_district_heating_areas
grid egon_map_zensus_district_heating_areas
grid egon_ehv_transfer_buses
grid osm_landuse
grid egon_hvmv_transfer_buses
grid egon_loadarea
grid osm_amenities_shops_filtered
grid egon_etrago_timeseries_individual_heating
grid branch_data
grid dcline_data
grid osm_buildings_residential
grid osm_amenities_in_buildings_filtered
grid osm_buildings_filtered_with_amenities
grid osm_amenities_not_in_buildings_filtered
grid osm_amenities_not_in_buildings_filtered
metadata datasets NO
metadata dependency_graph NO
openstreetmap osm_amenities_in_buildings_tmp NO
openstreetmap osm_amenities_not_in_buildings YES @nesnoj
openstreetmap osm_amenities_shops_filtered YES @nesnoj
openstreetmap osm_buildings YES @nesnoj
openstreetmap osm_buildings_filtered YES @nesnoj
openstreetmap osm_buildings_with_amenities YES @nesnoj
openstreetmap osm_buildings_with_res_tmp NO
openstreetmap osm_buildings_with_res_tmp2 NO
openstreetmap osm_landuse YES @IlkaCu
openstreetmap osm_line YES @nesnoj X
openstreetmap osm_nodes YES @nesnoj X
openstreetmap osm_point YES @nesnoj X
openstreetmap osm_polygon YES @nesnoj X
openstreetmap osm_rels YES @nesnoj X
openstreetmap osm_roads YES @nesnoj X
openstreetmap osm_ways YES @nesnoj X
openstreetmap osm_ways_preprocessed YES @nesnoj
openstreetmap osm_ways_with_segments YES @nesnoj
openstreetmap egon_building_heat_peak_loads
openstreetmap egon_ehv_substation
openstreetmap egon_ehv_substation_voronoi
openstreetmap egon_scenario_capacities
openstreetmap egon_nep_2021_conventional_powerplants
openstreetmap egon_mv_grid_district
openstreetmap egon_map_mvgriddistrict_vg250
openstreetmap iee_household_load_profiles
openstreetmap egon_destatis_zensus_household_per_ha_refined
openstreetmap egon_household_electricity_profile_in_census_cell
openstreetmap osm_buildings_synthetic YES
openstreetmap egon_household_electricity_profile_of_buildings
openstreetmap egon_saltstructures_storage_potential
openstreetmap egon_etrago_electricity_households
openstreetmap egon_demandregio_sites_ind_electricity
openstreetmap egon_demandregio_osm_ind_electricity
openstreetmap egon_osm_ind_load_curves
openstreetmap egon_osm_ind_load_curves_individual
osmtgmod_results branch_data NO
osmtgmod_results bus_data NO
osmtgmod_results dcline_data NO
osmtgmod_results results_metadata NO
osmtgmod_results egon_sites_ind_load_curves
osmtgmod_results egon_sites_ind_load_curves_individual
osmtgmod_results egon_demandregio_zensus_electricity
public _db_status NO
public abstr_values NO
public branch_data NO
public branch_specifications NO
public bus_data NO
public dcline_data NO
public dcline_specifications NO
public debug NO
public edit_power_relations NO
public nodes NO
public nuts_poly NO
public osm_metadata NO
public plz_poly NO
public power_circ_members NO
public power_circuits NO
public power_line NO
public power_line_sep NO
public power_relations NO
public power_substation NO
public power_ways NO
public problem_log NO
public relation_members NO
public relations NO
public schema_info NO
public spatial_ref_sys NO
public transfer_busses NO
public transfer_busses_complete NO
public transfer_busses_connect NO
public transfer_busses_connect_all NO
public transformer_specifications NO
public users NO
public way_nodes NO
public ways NO
scenario egon_scenario_parameters YES
scenario egon_etrago_electricity_cts_dsm_timeseries
society destatis_zensus_population_per_ha YES @IlkaCu
society destatis_zensus_population_per_ha_inside_germany YES @nesnoj X
society egon_demandregio_household NO
society egon_demandregio_population NO
society egon_destatis_zensus_apartment_building_population_per_ha YES @nesnoj
society egon_destatis_zensus_apartment_per_ha YES @IlkaCu
society egon_destatis_zensus_building_per_ha YES @IlkaCu
society egon_destatis_zensus_household_per_ha YES @IlkaCu
society egon_household_prognosis YES @IlkaCu
society egon_population_prognosis YES @IlkaCu
society egon_osm_ind_load_curves_individual_dsm_timeseries
society egon_etrago_ac_h2
society egon_etrago_ch4_h2
society egon_gas_voronoi
society egon_emob_charging_infrastructure YES
society egon_heavy_duty_transport_voronoi
society egon_chp_plants YES
society egon_mastr_conventional_without_chp
supply egon_chp YES @ClaraBuettner
supply egon_district_heating YES @ClaraBuettner
supply egon_era5_renewable_feedin YES @ClaraBuettner
supply egon_era5_weather_cells YES @ClaraBuettner
supply egon_individual_heating YES @ClaraBuettner
supply egon_mastr_conventional_without_chp NO
supply egon_power_plants YES @IlkaCu
supply egon_re_potential_area_pv_agriculture YES @nesnoj
supply egon_re_potential_area_pv_road_railway YES @nesnoj
supply egon_re_potential_area_wind YES @nesnoj
supply egon_scenario_capacities YES @ClaraBuettner
supply nep_2021_conv_powerplants NO
supply egon_mastr_pv_roof_geocoded
supply egon_power_plants_pv_roof_building YES
supply egon_storages
supply egon_home_batteries YES
supply egon_power_plants_wind YES
supply egon_power_plants_pv YES
supply egon_power_plants_biomass YES
supply egon_power_plants_hydro YES
@nesnoj nesnoj added 🙏 help wanted Extra attention is needed 🏗️ integration Integrating a data processing step labels Sep 22, 2021
@nesnoj
Copy link
Member Author

nesnoj commented Sep 24, 2021

Alright, now I'm all done with the metadata instructions and examples. Let's do some meta :)

@nesnoj
Copy link
Member Author

nesnoj commented Feb 28, 2023

We created some metadata 4 the HH paper which could be implemented

@ClaraBuettner
Copy link
Contributor

Since the table in the issue description is not that easy to edit and read, I created one on the cloud (1_Arbeitsinhalte/AP1/metadata/). I extracted the list of tables from one of the last CI-runs, so it should be up to date.
We will take care about moving all the entries from the table in this issue to the table. But not all rows are filled here, so we might need some help from you.

@birgits
Copy link
Contributor

birgits commented Mar 3, 2023

Thank you @ClaraBuettner :) Will you let us know, when entries are moved to the cloud table? Then we will continue filling it out there.

@nesnoj
Copy link
Member Author

nesnoj commented Mar 3, 2023

Some thoughts about table relations:
I think it'd be helpful to know to what other table the fields in resources fields refer to. A good place could be the field description.
Something like

            "schema": {
                "fields": [
                    {
                        "name": "zensus_population_id",
                        "description": "ID of Census cell, refers to field society.destatis_zensus_population_per_ha_inside_germany.id",
                        "type": "serial",
                        "unit": null
                    },

Or, if applicable, via the foreign key.
This makes it easier for users to join multiple tables (additional to example queries).

@ClaraBuettner
Copy link
Contributor

Thank you @ClaraBuettner :) Will you let us know, when entries are moved to the cloud table? Then we will continue filling it out there.

Yes, @Alonsoju95 will let you know.

@Alonsoju95
Copy link
Contributor

Alonsoju95 commented Mar 8, 2023

Hi, everyone!
All entries have been moved to the cloud table.
The link: https://wolke.rl-institut.de/s/9RyJKLP2HRGxdYf

@IlkaCu
Copy link
Member

IlkaCu commented Mar 14, 2023

I created a new tab 'Tables' in the mentioned document on the nextcloud, where I removed all the duplicates etc.

@khelfen
Copy link
Contributor

khelfen commented Mar 17, 2023

For the following tables I need the sources even if they are not in need of metadata:

demand.egon_etrago_electricity_cts
demand.egon_demandregio_sites_ind_electricity
demand.egon_osm_ind_load_curves_individual
demand.egon_sites_ind_load_curves_individual

nailend added a commit that referenced this issue Mar 17, 2023
ClaraBuettner added a commit that referenced this issue Mar 17, 2023
@khelfen khelfen linked a pull request Apr 14, 2023 that will close this issue
6 tasks
khelfen added a commit that referenced this issue Aug 20, 2024
khelfen added a commit that referenced this issue Aug 22, 2024
khelfen added a commit that referenced this issue Aug 22, 2024
khelfen added a commit that referenced this issue Aug 22, 2024
khelfen added a commit to openego/powerd-data that referenced this issue Aug 26, 2024
khelfen added a commit to openego/powerd-data that referenced this issue Oct 14, 2024
khelfen added a commit to openego/powerd-data that referenced this issue Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏗️ integration Integrating a data processing step 🙏 help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants