Data Input

Data input inside tessif is handled by the parse module. It provides read-in capabilities for processing numerous data types storing energy system information and transforms this information into a python manageable mapping.

Which utility to use depends on the input data format at hand. Usually recognizable via the filename extensions.

Concept

Inside tessif every bulk of logically grouped data is realised as mapping. The main concept of data input is no different. Hence all of the supported data formats represent a kind of mapping. These mappings are transformable by tessif’s parse functions.

All of them return a nested dictionairy (one of python’s native forms of mappings) with a pandas.DataFrame (one of python’s spreadsheet like objects) for each group of energy system component at it’s core.

Supported Data Formats

The following sections provides a brief overview of the supported input data formats as well as consequential links for more detailed information.

Note

The current support of the various input data formats (with the exception of Hardcoded Python Instance Objects) can be gauged and expanded using the tessif.parse module.

.cfg

Configuration file formats are simple yet powerful data formats for storing key-value pair based information (mappings). They are also editable by most operating systems and softwares. And as such they are an ideal candidate for storing energy system data for tessif, since they are easily read and edited by humans and machines alike.

The most common filename extensions are:

• cf

• cfg

• cnf

• conf

• ini

Tessif’s support for config files is two folded.

1. Flat Configuration Files

Flat configuration files in this context are understood as config files with only one type of energy system component. Meaning there is one file for Busses one for all the Sources, and so on. Additionally there is one specifying the timeframe.

Each component entity (e.g. my_bus_1, my_bus_2, …) is seperated by individual Sections. An example for this can be found in tessif.examples.data.tsf.cfg.flat.

Typical data/folder structures for storing energy system data in flat config files look like:

energy_systems
|
|-my_energy_system_1
| |-busses.cfg
| |-sinks.cfg
| |-storages.cfg
| |-sources.cfg
| |-timeframe.cfg
| |-transformers.cfg
|
|-my_energy_system_2
| |-busses.cfg
| |-sinks.cfg
| |-storages.cfg
| |-sources.cfg
| |-timeframe.cfg
| |-transformers.cfg

The advantages of using flat configuration files (as opposed to nested configuration files) are:

• Existance of a standard library parsing utility

• Data files are shorter/smaller since they are seperated into each energy system component by design.

Disadvantages are:

• Need of up to 6 files for storing one energy system and the subsequent file input/output streaming overhang

2. Nested Configuration Files

.hdf5

Hierarchical Data Format 5 (HDF5) is a data format designed to store and organize large amounts of data. One of its advantages is that it is possible to not load the complete file into the memory but only the needed parts of a file. HDF5 has a low latency and high throughput compared with other ways to store data. It is supported by many programming languages and operating systems.

Tessif expects HDF5 files to be written in such a way that there is a group for each kind of energy system component as well as one for the timeframe and one defining the global constraints. Each of these groups has a subgroup for each of that component’s entities which contains groups and datasets that store the parameters of that component.

An example can be found in tessif.examples.data.tsf.hdf5.

.json

.py

Python files are the most native in regard to tessif since it is written in python. It is therefore possible to hardcode energy system objects explicitly.

The support for energy system data represented in python files can be seperated into the following two major categories.

1. Hardcoded Python Instance Objects

Hardcoded python instance objects representing energy system components of the energy system simulation model to use are also a valid option for providing data. Most common use cases for this approach depending on the users python programing skill level are:

• Beginning and intermediate skilled python programers:

  • Already existing code due to exclusive use of the supported energy system simulation model prior to using tessif

  • Desire to get more familiar with tessif or the energy system simulation model of choice

• Advancedd python programers:

  • Monkey Patching the existing infrastructure

  • Automating the input data process generation utilizing instance objects directly

Examples can be found here (detailed) and here (subsections: ‘data/py - hardcoded’).

Once the energy system call instance has been coded, it can be accessed via (assuming the energy system is created using a function called CREATE_MY_ENERGY_SYSTEM an the corresponding module is found at PATH):

from PATH import CREATE_MY_ENERGY_SYSTEM

For the beforenamed example hub this would result in:

from tessif.examples.data.tsf.py_hard import create_fpwe
es = create_fpwe()

2. Pure Python Mappings

In addition to hardcoded python objects, energy systems can also be stored as python manageable mappings (or pure python mappings). Meaning every energy system can also be represented as nested dictionairy.

Most common use cases (again depending on the programers skill level) are:

• Beginning an intermediate skilled python programers:

  • Having 3rd party or custom written code/libraries returning dictionairies (since it’s one of the most commonly used throughout python projects)

  • Deepening the understanding of the transition from raw data formats to actual python obects using a nested dictionairy as a raw data set

  • Understanding how almost every energy system data set can be interpreted as mapping

• Advanced python programers:

  • Avoiding object creation overhang (mostly keyword characters) in huge datasets

  • Automating the input data process generation utilizing pure python mappings directly

Examples can be found here (subsections: ‘data/py - mapping’).

Once the energy system mapping has been coded it can accessed via (assuming the energy system mapping is stored in a variable called MY_ENERGY_SYSTEM_MAPPING of which the corresponding module can be found in PATH):

from PATH import MY_ENERGY_SYSTEM_MAPPING

It then needs to be parsed by tessif.parse.python_mapping() to ensure it is ready to be transformed from a mapping to an energy system:

import tessif.transform.mapping2es.tsf as tsf
import tessif.parse as parse

energy_system = tsf.transform(
    parse.python_mapping(MY_ENERGY_SYSTEM_MAPPING))

Where tsf can also be one of the supported energy system models.

For the beforenamed example hub this would result in:

from tessif.examples.data.tsf.py_mapping.fpwe import fpwe_mapping
from tessif.transform.mapping2es import tsf
import tessif.parse as parse

energy_system = tsf.transform(parse.python_mapping(fpwe_mapping))

Note

This approach also illustrates how all the other mappings of different data formats are processed.

.sdp

.xlsx - like

Within the context of tessif xlsx - like stands for xml based spreadsheet representing data like xlsx (Microsoft Excel) and ods (Open Document Spreadsheet).

Although not an ideal data format from a programers point of view, supporting spreadsheet like data representations yields following benefits:

• Decoupling data input and programing skills, which can be useful for:

  • Educational projects

  • Sharing workloads among teams and personel

  • Lowering the threshold for getting familiar with tessif

  • Providing backwards compatability to old-school engineering

• Enabling vba based preprocessing

• Building a bridge between open science and proprietary helpers

.xml

XML is a markup language to write data in a hierarchical structure. Its flexibility and its widespread use make it a viable option for storing tessifs energy system mappings.

Tessif expects xml files to be written in such a way that there is an element for each energy system component as well as one for the timeframe. Each element has a subelement for each entity of the component/timeframe. Its parameters are stored as the attributes of the subelement.

An example can be found in tessif.examples.data.tsf.xml.

.yaml