Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/#367 bpmn in docs #375

Open
wants to merge 15 commits into
base: dev
Choose a base branch
from
Open

Feature/#367 bpmn in docs #375

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
b0105b5
Use 100 % of page width for content
Sep 18, 2020
2fc392f
Reformat some data processing sections
Sep 18, 2020
b5460ec
Code formatting
Sep 18, 2020
f96de22
Use autosectionlabel in Sphinx
Sep 18, 2020
a6aeef3
Add BPMNs for data processing part to docs
Sep 18, 2020
4617c58
Re-organize Scripts page
Sep 18, 2020
f8e95b6
Reformat DP-pre BPMNs
Sep 19, 2020
b17c8d8
Add pre-processing BPMNs to docs
Sep 19, 2020
6418ffd
Update start page of documentation
Sep 19, 2020
a3a322b
Remove version restricty for numpy* in docs requirements
Sep 25, 2020
5c80154
Remove indices and tables
Sep 25, 2020
58e8022
Move content to landing page
Sep 25, 2020
9c9e607
Sort "Getting started" sectioons
Sep 25, 2020
392ddf9
Correct formatting
Sep 25, 2020
34640ed
Move content from README to RTD
Sep 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 1 addition & 76 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,86 +9,11 @@
eGo dataprocessing
===================

(geo)data processing, database setup, data validation
See the `documentation <https://data_processing.readthedocs.io>`_ hosted on ReadTheDocs.

=========
Copyleft
=========
Code licensed under "GNU Affero General Public License Version 3 (AGPL-3.0)"
It is a collaborative work with several copyright owner:
Cite as "eGo dataprocessing © Flensburg University of Applied Sciences, Centre for Sustainable Energy Systems © Europa-Universität Flensburg, Centre for Sustainable Energy Systems © Reiner Lemoine Institut © DLR Institute for Networked Energy Systems"

Installation
============

Installation of latest release

```
(sudo) pip3 install dataprocessing
```

Installation (of developer version) via pip on linux systems as follows

```
sudo pip3 install -e <path-to-data_processing-repo>
```

Troubleshooting

On Windows
***********

1. __Problem__: Installation of required package shapely fails. __Solution__: Install pre-build
wheel from [here](http://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely)


Run
====

You can run data processing by calling command-line script

```
~$ ego_data_processing
```

Local database with docker
==========================

You need to have `docker-compose` installed. And some more tools, see requirements_system.txt.

### Spin up database with docker

```
docker-compose up -d --build
```

### Database credentials

| database | dp |
| port | 54321 |
| user | oeuser |
| password | egon |


Usage
=====


Pre-processing
**************

Execute `eGo_import_and_preprocessing.py` for starting pre-processing from scratch doing data import and a pre-processing of imported tables.

Once completed, export required schemas to SQL dump

_You might have to install `postgresql-client-11` from additional source to have a compatible client with the v11 dockered PostgreSQL database._

```
/usr/lib/postgresql/11/bin/pg_dump -d dp -U oeuser -p 54321 -h localhost -Fc -n model_draft -n openstreetmap -n boundaries -n society > `date -I`_eGo_data_pre-processing_bremen.backup
```

Import a dump by

```
/usr/lib/postgresql/11/bin/pg_restore -d dp -U oeuser -p 54321 -h localhost 2020-03-28_eGo_data_pre-processing_deu.backup
```
13 changes: 6 additions & 7 deletions documentation/bpmn/ego_dp-pre_bpmn_section_cosmoclmgrid.graphml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@
<node id="n0" yfiles.foldertype="group">
<data key="d6">
<y:TableNode configuration="com.yworks.bpmn.Pool">
<y:Geometry height="797.6508900133493" width="1035.253596296578" x="2.1972465072707337" y="-59.0723684210526"/>
<y:Geometry height="728.6508900133493" width="731.253596296578" x="2.1972465072707337" y="-59.0723684210526"/>
<y:Fill color="#3265FF7F" color2="#D4D4D4CC" transparent="false"/>
<y:BorderStyle color="#000000" type="line" width="3.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" borderDistance="0.0" fontFamily="Dialog" fontSize="36" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" height="45.90625" horizontalTextPosition="center" iconTextGap="4" modelName="internal" modelPosition="l" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="722.5234375" x="0.0" xml:space="preserve" y="37.56372625667461">eGo Pre-Processing - cosmoclmgrid</y:NodeLabel>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="35" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" hasText="false" height="4.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="4.0" x="53.0" y="408.9875543816748">
<y:NodeLabel alignment="center" autoSizePolicy="content" borderDistance="0.0" fontFamily="Dialog" fontSize="36" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" height="45.90625" horizontalTextPosition="center" iconTextGap="4" modelName="internal" modelPosition="l" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="722.5234375" x="0.0" xml:space="preserve" y="3.063726256674613">eGo Pre-Processing - cosmoclmgrid</y:NodeLabel>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="35" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" hasText="false" height="4.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="4.0" x="53.0" y="374.4875543816748">
<y:LabelModel>
<y:RowNodeLabelModel offset="3.0"/>
</y:LabelModel>
Expand Down Expand Up @@ -53,18 +53,18 @@
</y:StyleProperties>
<y:State autoResize="true" closed="false" closedHeight="80.0" closedWidth="100.0"/>
<y:Insets bottom="0" bottomF="0.0" left="0" leftF="0.0" right="0" rightF="0.0" top="0" topF="0.0"/>
<y:BorderInsets bottom="246" bottomF="245.51742808101972" left="70" leftF="69.50606340184152" right="400" rightF="399.74753289473665" top="165" topF="164.89450998185123"/>
<y:BorderInsets bottom="177" bottomF="176.51742808101972" left="70" leftF="69.50606340184152" right="96" rightF="95.74753289473654" top="165" topF="164.89450998185123"/>
<y:Table autoResizeTable="true" defaultColumnWidth="360.0" defaultMinimumColumnWidth="30.0" defaultMinimumRowHeight="30.0" defaultRowHeight="107.0">
<y:DefaultColumnInsets bottom="3.0" left="3.0" right="3.0" top="20.0"/>
<y:DefaultRowInsets bottom="3.0" left="20.0" right="3.0" top="3.0"/>
<y:Insets bottom="0.0" left="50.0" right="0.0" top="0.0"/>
<y:Columns>
<y:Column id="column_0" minimumWidth="30.0" width="961.573908796578">
<y:Column id="column_0" minimumWidth="30.0" width="657.573908796578">
<y:Insets bottom="3.0" left="3.0" right="3.0" top="20.0"/>
</y:Column>
</y:Columns>
<y:Rows>
<y:Row height="781.9751087633496" id="row_0" minimumHeight="30.0">
<y:Row height="712.9751087633496" id="row_0" minimumHeight="30.0">
<y:Insets bottom="3.0" left="20.0" right="3.0" top="3.0"/>
</y:Row>
</y:Rows>
Expand Down Expand Up @@ -255,7 +255,6 @@
</data>
</edge>
<edge id="n0::e4" source="n0::n0" target="n0::n5">
<data key="d9"/>
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0"/>
Expand Down
10 changes: 5 additions & 5 deletions documentation/bpmn/ego_dp-pre_bpmn_section_demand_federalstate.graphml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@
<node id="n0" yfiles.foldertype="group">
<data key="d6">
<y:TableNode configuration="com.yworks.bpmn.Pool">
<y:Geometry height="704.8630610659811" width="1031.235947258848" x="-39.441426932992954" y="181.68421052631584"/>
<y:Geometry height="904.8630610659811" width="1031.235947258848" x="-39.441426932992954" y="67.68421052631584"/>
<y:Fill color="#3265FF7F" color2="#D4D4D4CC" transparent="false"/>
<y:BorderStyle color="#000000" type="line" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" borderDistance="0.0" fontFamily="Dialog" fontSize="36" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" height="45.90625" horizontalTextPosition="center" iconTextGap="4" modelName="internal" modelPosition="l" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="899.04296875" x="0.0" xml:space="preserve" y="-97.08995384200944">eGo Pre-Processing - demand federal states</y:NodeLabel>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="35" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" hasText="false" height="4.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="4.0" x="53.0" y="362.5936399079907">
<y:NodeLabel alignment="center" autoSizePolicy="content" borderDistance="0.0" fontFamily="Dialog" fontSize="36" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" height="45.90625" horizontalTextPosition="center" iconTextGap="4" modelName="internal" modelPosition="l" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="899.04296875" x="0.0" xml:space="preserve" y="2.910046157990564">eGo Pre-Processing - demand federal states</y:NodeLabel>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="35" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" hasText="false" height="4.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="4.0" x="53.0" y="462.5936399079907">
<y:LabelModel>
<y:RowNodeLabelModel offset="3.0"/>
</y:LabelModel>
Expand Down Expand Up @@ -53,7 +53,7 @@
</y:StyleProperties>
<y:State autoResize="true" closed="false" closedHeight="80.0" closedWidth="100.0"/>
<y:Insets bottom="0" bottomF="0.0" left="0" leftF="0.0" right="0" rightF="0.0" top="0" topF="0.0"/>
<y:BorderInsets bottom="176" bottomF="176.07852159229697" left="0" leftF="0.0" right="124" rightF="124.45801526717537" top="128" topF="127.92418642024904"/>
<y:BorderInsets bottom="262" bottomF="262.07852159229697" left="0" leftF="0.0" right="124" rightF="124.45801526717537" top="242" topF="241.92418642024904"/>
<y:Table autoResizeTable="true" defaultColumnWidth="360.0" defaultMinimumColumnWidth="30.0" defaultMinimumRowHeight="30.0" defaultRowHeight="107.0">
<y:DefaultColumnInsets bottom="3.0" left="3.0" right="3.0" top="20.0"/>
<y:DefaultRowInsets bottom="3.0" left="20.0" right="3.0" top="3.0"/>
Expand All @@ -64,7 +64,7 @@
</y:Column>
</y:Columns>
<y:Rows>
<y:Row height="689.1872798159814" id="row_0" minimumHeight="30.0">
<y:Row height="889.1872798159814" id="row_0" minimumHeight="30.0">
<y:Insets bottom="3.0" left="20.0" right="3.0" top="3.0"/>
</y:Row>
</y:Rows>
Expand Down
25 changes: 12 additions & 13 deletions documentation/bpmn/ego_dp-pre_bpmn_section_osmtgmod.graphml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@
<node id="n0" yfiles.foldertype="group">
<data key="d6">
<y:TableNode configuration="com.yworks.bpmn.Pool">
<y:Geometry height="956.0380671954874" width="852.7382574364797" x="-21.679684369472625" y="157.55921052631584"/>
<y:Geometry height="675.778381331613" width="754.8534406825529" x="-21.679684369472625" y="328.60004822265086"/>
<y:Fill color="#3265FF7F" color2="#D4D4D4CC" transparent="false"/>
<y:BorderStyle color="#000000" type="line" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" borderDistance="0.0" fontFamily="Dialog" fontSize="36" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" height="45.90625" horizontalTextPosition="center" iconTextGap="4" modelName="internal" modelPosition="l" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="665.74609375" x="0.0" xml:space="preserve" y="145.14598672274371">eGo Pre-Processing - osmTGmod</y:NodeLabel>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="35" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" hasText="false" height="4.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="4.0" x="53.0" y="488.1811429727438">
<y:NodeLabel alignment="center" autoSizePolicy="content" borderDistance="0.0" fontFamily="Dialog" fontSize="36" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" height="45.90625" horizontalTextPosition="center" iconTextGap="4" modelName="internal" modelPosition="l" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="665.74609375" x="0.0" xml:space="preserve" y="5.016143790806382">eGo Pre-Processing - osmTGmod</y:NodeLabel>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="35" fontStyle="bold" hasBackgroundColor="false" hasLineColor="false" hasText="false" height="4.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" rotationAngle="270.0" textColor="#000000" verticalTextPosition="bottom" visible="true" width="4.0" x="53.0" y="348.05130004080667">
<y:LabelModel>
<y:RowNodeLabelModel offset="3.0"/>
</y:LabelModel>
Expand All @@ -31,40 +31,40 @@
</y:NodeLabel>
<y:StyleProperties>
<y:Property class="java.awt.Color" name="com.yworks.bpmn.icon.line.color" value="#000000"/>
<y:Property name="y.view.tabular.TableNodePainter.ALTERNATE_ROW_STYLE">
<y:SimpleStyle fillColor="#474A4340" lineColor="#000000" lineType="line" lineWidth="1.0"/>
</y:Property>
<y:Property name="y.view.tabular.TableNodePainter.ALTERNATE_COLUMN_SELECTION_STYLE">
<y:SimpleStyle fillColor="#474A4380" lineColor="#000000" lineType="line" lineWidth="3.0"/>
</y:Property>
<y:Property name="y.view.tabular.TableNodePainter.ALTERNATE_ROW_SELECTION_STYLE">
<y:SimpleStyle fillColor="#474A4380" lineColor="#000000" lineType="line" lineWidth="3.0"/>
</y:Property>
<y:Property class="java.awt.Color" name="POOL_LANE_COLOR_MAIN" value="#ffffffbf"/>
<y:Property class="com.yworks.yfiles.bpmn.view.BPMNTypeEnum" name="com.yworks.bpmn.type" value="POOL_TYPE_LANE"/>
<y:Property name="y.view.tabular.TableNodePainter.ALTERNATE_ROW_STYLE">
<y:SimpleStyle fillColor="#474A4340" lineColor="#000000" lineType="line" lineWidth="1.0"/>
</y:Property>
<y:Property class="java.awt.Color" name="POOL_LANE_COLOR_ALTERNATING" value="#ffffffbf"/>
<y:Property class="java.awt.Color" name="POOL_LANE_COLOR_MAIN" value="#ffffffbf"/>
<y:Property name="POOL_HEADER_COLOR_MAIN"/>
<y:Property class="java.lang.String" name="POOL_LANE_STYLE" value="LANE_STYLE_COLUMNS"/>
<y:Property class="java.awt.Color" name="com.yworks.bpmn.icon.fill2" value="#d4d4d4cc"/>
<y:Property class="java.awt.Color" name="com.yworks.bpmn.icon.fill" value="#ffffffe6"/>
<y:Property class="com.yworks.yfiles.bpmn.view.BPMNTypeEnum" name="com.yworks.bpmn.type" value="POOL_TYPE_LANE"/>
<y:Property name="y.view.tabular.TableNodePainter.ALTERNATE_COLUMN_STYLE">
<y:SimpleStyle fillColor="#474A4340" lineColor="#000000" lineType="line" lineWidth="1.0"/>
</y:Property>
</y:StyleProperties>
<y:State autoResize="true" closed="false" closedHeight="80.0" closedWidth="100.0"/>
<y:Insets bottom="0" bottomF="0.0" left="0" leftF="0.0" right="0" rightF="0.0" top="0" topF="0.0"/>
<y:BorderInsets bottom="254" bottomF="253.81414473684208" left="38" leftF="38.23825743647967" right="173" rightF="172.5" top="242" topF="241.53265648872298"/>
<y:BorderInsets bottom="145" bottomF="144.59529656930272" left="38" leftF="38.23825743647967" right="75" rightF="74.61518324607323" top="70" topF="70.49181879238796"/>
<y:Table autoResizeTable="true" defaultColumnWidth="360.0" defaultMinimumColumnWidth="30.0" defaultMinimumRowHeight="30.0" defaultRowHeight="107.0">
<y:DefaultColumnInsets bottom="3.0" left="3.0" right="3.0" top="20.0"/>
<y:DefaultRowInsets bottom="3.0" left="20.0" right="3.0" top="3.0"/>
<y:Insets bottom="0.0" left="50.0" right="0.0" top="0.0"/>
<y:Columns>
<y:Column id="column_0" minimumWidth="30.0" width="779.0585699364797">
<y:Column id="column_0" minimumWidth="30.0" width="681.1737531825529">
<y:Insets bottom="3.0" left="3.0" right="3.0" top="20.0"/>
</y:Column>
</y:Columns>
<y:Rows>
<y:Row height="940.3622859454877" id="row_0" minimumHeight="30.0">
<y:Row height="660.1026000816132" id="row_0" minimumHeight="30.0">
<y:Insets bottom="3.0" left="20.0" right="3.0" top="3.0"/>
</y:Row>
</y:Rows>
Expand Down Expand Up @@ -167,10 +167,10 @@ grid-otg_ehvhv_results_metadata<y:LabelModel><y:SmartNodeLabelModel distance="4.
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="17.96875" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="78.87109375" x="-21.935546875" xml:space="preserve" y="59.0">OSM Import <y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="-0.5" nodeRatioX="0.0" nodeRatioY="0.5" offsetX="0.0" offsetY="4.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
<y:StyleProperties>
<y:Property class="java.awt.Color" name="com.yworks.bpmn.icon.line.color" value="#000000"/>
<y:Property class="com.yworks.yfiles.bpmn.view.DataObjectTypeEnum" name="com.yworks.bpmn.dataObjectType" value="DATA_OBJECT_TYPE_INPUT"/>
<y:Property class="java.awt.Color" name="com.yworks.bpmn.icon.fill2" value="#d4d4d4cc"/>
<y:Property class="java.awt.Color" name="com.yworks.bpmn.icon.fill" value="#ffffffe6"/>
<y:Property class="com.yworks.yfiles.bpmn.view.BPMNTypeEnum" name="com.yworks.bpmn.type" value="ARTIFACT_TYPE_DATA_OBJECT"/>
<y:Property class="com.yworks.yfiles.bpmn.view.DataObjectTypeEnum" name="com.yworks.bpmn.dataObjectType" value="DATA_OBJECT_TYPE_INPUT"/>
</y:StyleProperties>
</y:GenericNode>
</data>
Expand Down Expand Up @@ -219,7 +219,6 @@ grid-otg_ehvhv_results_metadata<y:LabelModel><y:SmartNodeLabelModel distance="4.
</data>
</edge>
<edge id="n0::e3" source="n0::n2" target="n0::n1">
<data key="d9"/>
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0"/>
Expand Down
Loading