412 lines
20 KiB
HTML
Executable File
412 lines
20 KiB
HTML
Executable File
<!doctype html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
|
|
<title>Slides</title>
|
|
<link rel="stylesheet" href="dist/reset.css">
|
|
<link rel="stylesheet" href="dist/reveal.css">
|
|
<link rel="stylesheet" href="dist/theme/white.css">
|
|
<link rel="stylesheet" href="plugin/highlight/monokai.css">
|
|
<style>
|
|
.presentation-footer {
|
|
position: fixed;
|
|
bottom: 12px;
|
|
left: 0;
|
|
width: 100%;
|
|
|
|
display: flex;
|
|
justify-content: space-between;
|
|
align-items: center;
|
|
|
|
padding: 0 16px;
|
|
box-sizing: border-box;
|
|
|
|
font-size: 24px;
|
|
color: #004682;
|
|
|
|
z-index: 1000;
|
|
pointer-events: none;
|
|
|
|
font-family: sans-serif;
|
|
}
|
|
|
|
.hljs {
|
|
font-family: monospace !important;
|
|
}
|
|
.reveal h3 {
|
|
font-size: 32px;
|
|
}
|
|
th, td {
|
|
font-size:24px;
|
|
}
|
|
li {
|
|
font-size: 28px;
|
|
}
|
|
cite {
|
|
font-style: normal;
|
|
font-size: 0.5em;
|
|
color: #6b7280;
|
|
background: #f3f4f6;
|
|
padding: 0.05em 0.25em;
|
|
border-radius: 0.25em;
|
|
margin-left: 0.15em;
|
|
white-space: nowrap;
|
|
}
|
|
.inline_cite {
|
|
display: inline;
|
|
font-style: normal;
|
|
font-size: 0.5em;
|
|
color: #6b7280;
|
|
background: #f3f4f6;
|
|
padding: 0.1em 0.35em;
|
|
border-radius: 0.25em;
|
|
margin-left: 0.15em;
|
|
white-space: nowrap;
|
|
}
|
|
p {
|
|
font-size: 28px;
|
|
}
|
|
b {
|
|
color: #004682;
|
|
}
|
|
.reveal .slide-number {
|
|
font-size: 24px;
|
|
background: none;
|
|
color: #004682;
|
|
|
|
padding: 0 16px;
|
|
}
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<div class="reveal">
|
|
<div class="slides">
|
|
<section data-background-image="images/solarparc.jpg" data-background-opacity="0.40"><h3>Building Pipelines for the Solar Industry</h3>
|
|
<p>Internship Report</p>
|
|
<hr>
|
|
<small>Joaquin Gottlebe</small>
|
|
<small class="date"></small>
|
|
<br>
|
|
<img src="https://www.enerparc.de/img/static/logo_enerparc.png" width="200px"></img>
|
|
</section>
|
|
<section><h3>Geographic Information (Geodata)</h3>
|
|
<div class="r-stack">
|
|
|
|
<img class="fragment fade-out r-stretch" src="images/geodata_features.png"></img>
|
|
|
|
<table class="fragment current-visible" border="1" cellpadding="6" cellspacing="0">
|
|
<thead>
|
|
<tr>
|
|
<th>id</th>
|
|
<th>name</th>
|
|
<th>category</th>
|
|
<th>elevation_m</th>
|
|
<th>geometry</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td>1</td>
|
|
<td>Berlin Center</td>
|
|
<td>city</td>
|
|
<td>34</td>
|
|
<td>POINT(13.4050 52.5200)</td>
|
|
</tr>
|
|
<tr>
|
|
<td>2</td>
|
|
<td>Potsdam Station</td>
|
|
<td>transport</td>
|
|
<td>38</td>
|
|
<td>POINT(13.0657 52.3914)</td>
|
|
</tr>
|
|
<tr>
|
|
<td>3</td>
|
|
<td>Forest Plot A</td>
|
|
<td>sample_site</td>
|
|
<td>72</td>
|
|
<td>POINT(12.9876 52.4123)</td>
|
|
</tr>
|
|
<tr>
|
|
<td>4</td>
|
|
<td>Lake View</td>
|
|
<td>observation</td>
|
|
<td>29</td>
|
|
<td>POINT(13.1025 52.4568)</td>
|
|
</tr>
|
|
<tr>
|
|
<td>5</td>
|
|
<td>Weather Sensor 01</td>
|
|
<td>sensor</td>
|
|
<td>45</td>
|
|
<td>POINT(13.2211 52.3347)</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p><b>Definition: </b>[...] Information about locations on or near the surface of the Earth [...]"</p>
|
|
<p><cite>michaelminn.net/tutorials/arcgis-pro-terrain/</cite> <cite>Liu & Özsu, 2009: Encyclopedia of Database Systems</cite></p>
|
|
</section>
|
|
<section><h3>Basics: Spatial Operations</h3>
|
|
<img src="https://geopandas.org/en/stable/_images/overlay_operations.png" ></img>
|
|
<p><b>Definition: </b>(Spatial) operations refer to the operations that an end user
|
|
performs on maps stored in a database.</p>
|
|
<p><cite>geopandas.org</cite> <cite>Liu & Özsu, 2009: Encyclopedia of Database Systems</cite></p>
|
|
</section>
|
|
<section><h3>Basics: Geographic Information Systems (GIS)</h3>
|
|
<img src="images/qgis.png" class="r-stretch"></img>
|
|
<p><b>Definition: </b>"[...] is a computer application designed to perform a wide range of operations on geographic information. [...] A GIS includes functions to input, store, visualize, export, and analyze (geographic) information."</p>
|
|
<p><cite>©OpenStreetMap contributors</cite> <cite>Liu & Özsu, 2009: Encyclopedia of Database Systems</cite></p>
|
|
</section>
|
|
<section><h3>Internship Overview</h3>
|
|
<p><b>Company: </b>Enerparc AG</p>
|
|
<p><b>Industry domain: </b>Solar industry</p>
|
|
<p><b>Duration: </b>360 hours over 4 months</p>
|
|
<p><b>Role: </b>GIS Developer</p>
|
|
</section>
|
|
<section><h3>My Background</h3>
|
|
<p><b>Bachelor in Geography: </b>Specialised in Remote Sensing and Geographic Information Systems and had 4 years of experience tutoring RS/GIS</p>
|
|
<p><b>Master in Computational Science: </b>Currently pursuing, in my 4th semester</p>
|
|
</section>
|
|
<section><h3>Why Enerparc</h3>
|
|
<p>First industry <b>experience</b></p>
|
|
<p>How is the dev experience in a <b>non-tech company</b>?</p>
|
|
</section>
|
|
<section><h5>Enerparc Expertise</h5>
|
|
<p>The company is involved in every stage of <b>solar energy projects</b>, acting as both a <b>project developer, operator and service provider</b> for third parties.</p>
|
|
<img src="images/enerparc_expertise.png" class="r-stretch"></img>
|
|
<p><cite>ENERPARC AG. enerparc.de</cite></p>
|
|
</section>
|
|
<section><h5>Enerparc in numbers</h5>
|
|
<img src="images/enerparc_overview.png" class="r-stretch"></img>
|
|
<p>Covers the electricity needs of <b>1.2 million households</b></p>
|
|
<p><cite>ENERPARC AG. enerparc.de</cite></p>
|
|
</section>
|
|
<section><h5>Enerparc Solar Parcs in Germany</h5>
|
|
<iframe
|
|
src="maps/solar_enerparc.html"
|
|
style="width: 100%; height: 480px; border: 0;"
|
|
loading="lazy">
|
|
</iframe>
|
|
<p><cite>ENERPARC AG. enerparc.de/en/projects#solar-parks-germany</cite></p>
|
|
</section>
|
|
<section><h3>My Team</h3>
|
|
<p><b>Department: </b>GIS Development & Administration</p>
|
|
<p><b>Personnel: </b>1 Team lead, 1 Full time, 2 Working students</p>
|
|
<p><b>Tasks: </b>GIS Plugin development, Geodata engineering, WebGIS administration, Geodatabase administration</p>
|
|
</section>
|
|
<section><h3>Project</h3>
|
|
<p><b>Research question: </b>How to handle geodata pipelines?</p>
|
|
<p><b>Needed: </b>Daily updated geodata on energy assets of Germany</p>
|
|
<p><b>Use cases: </b>Monitoring of market activity</p>
|
|
<p><b>Architecture: </b>ETL (extract, transform, load)</p>
|
|
</section>
|
|
<section><h3>ETL Pipeline</h3>
|
|
<img src="images/etl_flow.svg" height="500px"></img>
|
|
<p><cite>own illustration</cite></p>
|
|
</section>
|
|
<section><h3>Extraction</h3>
|
|
<img src="images/etl_flow_extraction.svg" height="500px"></img>
|
|
<p><cite>own illustration</cite></p>
|
|
</section>
|
|
<section><h3>Extraction: Marktstammdatenregister</h3>
|
|
<img src="https://www.marktstammdatenregister.de/MaStR/Assets/img/MaStR_Logo.svg" width="600px"></img>
|
|
<p><b>Description: </b>Official registry of all facilities and units in the German energy system</p>
|
|
<p><b>Provider: </b>Bundesnetzagentur</p>
|
|
<p><b>Access Methods: </b>SOAP API (Simple Object Access Protocol) / XML bulk download</p>
|
|
<p><cite>marktstammdatenregister.de</cite></p>
|
|
</section>
|
|
<section><h3>Extraction: Marktstammdatenregister</h3>
|
|
<img src="images/rdm.png" width="800px"></img>
|
|
<p><cite>marktstammdatenregister.de</cite></p>
|
|
</section>
|
|
<section><h3>Extraction: Marktstammdatenregister</h3>
|
|
<p><b>Development:</b></p>
|
|
<ol>
|
|
<li class="fragment">Tried out the SOAP API</li>
|
|
<li class="fragment">Realised an API is made for small extractions</li>
|
|
<li class="fragment">Tried out the bulk download with hundreds of XML files</li>
|
|
<li class="fragment">Realised the XML's are relational to each other and encoded</li>
|
|
<li class="fragment">Wrote a whole library to handle the XML dump</li>
|
|
<li class="fragment"><b>Found out there is already an open source library for this (open-mastr)</b></li>
|
|
</ol>
|
|
</section>
|
|
<section><h3>Extraction: Enerparc Assets</h3>
|
|
<p><b>Description: </b>Current photovoltaic systems owned by Enerparc AG</p>
|
|
<p><b>Provider: </b>Enerparc AG Engineering</p>
|
|
<p><b>Access Methods: </b>REST API (Representational State Transfer)</p>
|
|
</section>
|
|
<section><h3>Extraction: Enerparc Assets</h3>
|
|
<p><b>Development:</b></p>
|
|
<ol>
|
|
<li class="fragment">Asked the engineering department for the list of IDs</li>
|
|
<li class="fragment">Realised they always change</li>
|
|
<li class="fragment">Asked the engineering department for access to the DB</li>
|
|
<li class="fragment">They created a new REST API endpoint for this</li>
|
|
<li class="fragment"><b>Accessed this endpoint over a simple http request</b></li>
|
|
</ol>
|
|
</section>
|
|
<section><h3>Extraction: Borders</h3>
|
|
<p><b>Description: </b>Country & Municipality Borders</p>
|
|
<p><b>Provider: </b>Enerparc AG GIS</p>
|
|
<p><b>Access Methods: </b>SQL Queries</p>
|
|
<p><b>Database Model: </b>Relational</p>
|
|
</section>
|
|
<section><h3>Extraction: Borders</h3>
|
|
<p><b>Development:</b></p>
|
|
<ol>
|
|
<li class="fragment">Added the data manually to the repository</li>
|
|
<li class="fragment">Realised borders can also change</li>
|
|
<li class="fragment"><b>Accessed them using SQL statements during the processing</b></li>
|
|
</ol>
|
|
</section>
|
|
<section><h3>Transformation</h3>
|
|
<img src="images/etl_flow_transformation.svg" height="500px"></img>
|
|
<p><cite>own illustration</cite></p>
|
|
</section>
|
|
<section><h3>Transformation</h3>
|
|
<p><b>Development:</b></p>
|
|
<ol>
|
|
<li class="fragment">Started with Python and GeoPandas (GeoDataFrames)</li>
|
|
<li class="fragment">Unknowingly reinvented SQL in Python</li>
|
|
<li class="fragment">Realised SQL is way better for stuff like this</li>
|
|
<li class="fragment">Used a local SQLite DB for processing</li>
|
|
<li class="fragment">But still needed to do some spatial operations in python</li>
|
|
<li class="fragment">Realised the destination server has PostGIS (SQL for spatial operations)</li>
|
|
<li class="fragment">Moved whole pipeline into the database with some absurdly long SQL aka "I replaced my entire stack with PostgreSQL"</li>
|
|
<li class="fragment">Processing took over a day and crashed the server</li>
|
|
<li class="fragment">Realised the database server only has 4 GB RAM and a weak CPU</li>
|
|
<li class="fragment"><b>Moved everything back to Python</b></li>
|
|
<li class="fragment"><b>They lived happily ever after</b></li>
|
|
</ol>
|
|
</section>
|
|
<section><h3>Loading</h3>
|
|
<img src="images/etl_flow_loading.svg" height="500px"></img>
|
|
<p><cite>own illustration</cite></p>
|
|
</section>
|
|
<section><h3>Loading</h3>
|
|
<p><b>Development:</b></p>
|
|
<ol>
|
|
<li class="fragment">Used GeoPandas built in upload functionality</li>
|
|
<li class="fragment">Found out open-mastr can load it directly to the PostgreSQL database</li>
|
|
<li class="fragment">Since I moved everything to the DB anyway, this was fitting</li>
|
|
<li class="fragment"><b>But then again I moved everything back</b></li>
|
|
</ol>
|
|
</section>
|
|
<section><h3>Architecture</h3>
|
|
<img src="images/architecture.svg" height="525px"></img>
|
|
<p><cite>own illustration</cite></p>
|
|
</section>
|
|
<section><h5>Results: Photovoltaic Systems</h5>
|
|
<iframe
|
|
src="maps/solar.html"
|
|
style="width: 100%; height: 480px; border: 0;"
|
|
loading="lazy">
|
|
</iframe>
|
|
<p><cite>Data: Marktstammdatenregister</cite></p>
|
|
</section>
|
|
<section><h5>Results: Battery Storage Systems</h5>
|
|
<iframe
|
|
src="maps/battery.html"
|
|
style="width: 100%; height: 480px; border: 0;"
|
|
loading="lazy">
|
|
</iframe>
|
|
<p><cite>Data: Marktstammdatenregister</cite></p>
|
|
</section>
|
|
<section><h5>Results: Wind Turbines</h5>
|
|
<iframe
|
|
src="maps/wind.html"
|
|
style="width: 100%; height: 480px; border: 0;"
|
|
loading="lazy">
|
|
</iframe>
|
|
<p><cite>Data: Marktstammdatenregister</cite></p>
|
|
</section>
|
|
<section><h3>Reflections: Building ETL Pipelines</h3>
|
|
<ul>
|
|
<li>
|
|
<p>"Those who do not understand SQL are condemned to reinvent it, poorly." <span id="inline_cite">Adapted from Henry Spencer</span></p>
|
|
</li>
|
|
<li>
|
|
<p>Check your system specifications before you change architectures.</p>
|
|
</li>
|
|
<li>There is still a lot to learn ...</li>
|
|
</ul>
|
|
</section>
|
|
<section><h3>Reflections: First Industry Experience</h3>
|
|
<p><b>Positive</b></p>
|
|
<ul>
|
|
<li>Reduced pressure to perform</li>
|
|
<li>Satisfaction of seeing your creation in production</li>
|
|
<li>Obstkorb</li>
|
|
</ul>
|
|
<p><b>Negative</b></p>
|
|
<ul>
|
|
<li>A lot of meetings</li>
|
|
<li>Slow processes</li>
|
|
<li>Rigid structures</li>
|
|
</ul>
|
|
</section>
|
|
<section><h3>Reflections: Dev Experience in a Non-Tech Company</h3>
|
|
<p><b>Positive</b></p>
|
|
<ul>
|
|
<li>You can learn a lot</li>
|
|
<li>More independence</li>
|
|
<li>More control over your projects</li>
|
|
<li>Access to more important projects</li>
|
|
</ul>
|
|
<p><b>Negative</b></p>
|
|
<ul>
|
|
<li>More responsibilities</li>
|
|
<li>More task switching</li>
|
|
<li>Less supervision</li>
|
|
</ul>
|
|
</section>
|
|
<section><h3>Next Project</h3>
|
|
<p><b>Geo-ETL library for WFS Services</b></p>
|
|
<ul>
|
|
<li><b>Amtliches Liegenschaftskatasterinformationssystem</b> (Gemarkungen, Flure, Flurstuecke, Gebaeudedaten, ...)</li>
|
|
<li><b>Bundesamt fuer Naturschutz</b> (Vogelschutzgebiete, Nationalparke, Naturschutzgebiete,...)</li>
|
|
<li><b>Biotope</b></li>
|
|
<li><b>...</b></li>
|
|
</ul>
|
|
<br></br>
|
|
<small>WFS (Web Feature Service): This standard defines direct fine-grained access to geographic information <span class="inline_cite">ogc.org/standards/wfs/</span></small>
|
|
</section>
|
|
<section><h3>Acknowledgments</h3>
|
|
<p>Thanks to my team and Enerparc AG.</p>
|
|
|
|
<img src="https://www.enerparc.de/img/static/logo_enerparc.png" width="200px"></img>
|
|
<p>And thank you for listening.</p>
|
|
</section>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="presentation-footer">
|
|
<span class="footer-left">Joaquin Gottlebe / Building Pipelines for the Solar Industry / <span class="date"></span></span>
|
|
<span class="footer-right" id="footer-date"></span>
|
|
</div>
|
|
|
|
<script src="dist/reveal.js"></script>
|
|
<script src="plugin/notes/notes.js"></script>
|
|
<script src="plugin/markdown/markdown.js"></script>
|
|
<script src="plugin/highlight/highlight.js"></script>
|
|
<script src="plugin/title-footer/title-footer.js"></script>
|
|
<script>
|
|
Reveal.initialize({
|
|
controls: false,
|
|
transition: 'fade',
|
|
slideNumber: 'c/t',
|
|
hash: true,
|
|
plugins: [ RevealMarkdown, RevealHighlight, RevealNotes ]
|
|
});
|
|
</script>
|
|
<script>
|
|
const formattedDate = new Date().toLocaleDateString("de-DE");
|
|
|
|
document.querySelectorAll(".date").forEach(el => {
|
|
el.textContent = formattedDate;
|
|
});
|
|
</script>
|
|
</body>
|
|
</html>
|