Product
Introducing License Enforcement in Socket
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Python scripts for FAME models, generation of protobuf input files and conversion of protobuf output files. Please visit the FAME-Wiki to get an explanation of FAME and its components.
We recommend installing fameio
using PyPI:
pip install fameio
You may also use pipx
. For detailed information please refer to the official pipx
documentation.
pipx install fameio
fameio
is currently developed and tested for Python 3.8 or higher.
See the pyproject.toml
for a complete listing of dependencies.
FAME-Io currently offers two main scripts makeFameRunConfig
and convertFameResults
.
Both are automatically installed with the package.
The first one creates a protobuf file for FAME applications using YAML definition files and CSV files.
The latter one reads output files from FAME applications in protobuf format and converts them to CSV files.
You may use the example data provided for the AMIRIS model which can be used to simulate electricity markets in Germany, Austria, and a simple proof-of-concept model.
Digests configuration files in YAML format, combines them with CSV data files and creates a single input file for FAME applications in protobuf format. Call structure:
makeFameRunConfig -f <path/to/scenario.yaml>
You may also specify any of the following arguments:
Command | Action |
---|---|
-l or --log | Sets the logging level. Default is info . Options are debug , info , warning , warn , error , critical . |
-lf or --logfile | Sets the logging file. Default is None . If None is provided, all logs get only printed to the console. |
-o or --output | Sets the path of the compiled protobuf output file. Default is config.pb . |
-enc or --encoding | Sets the encoding of all yaml files to the given one (e.g. 'utf8' or 'cp1252'. Default is None , i.e. your operating system's standard. |
This could look as follows:
makeFameRunConfig -f <path/to/scenario.yaml> -l debug -lf <path/to/scenario.log> -o <path/to/config.pb>
You may also call the configuration builder from any Python script with
from fameio.scripts.make_config import Options, run as make_config
make_config({Options.FILE: "path/to/scenario.yaml", })
Similar to the console call you may also specify custom run config arguments and add it in a dictionary to the function call.
from fameio.scripts.make_config import Options, run as make_config
run_config = {Options.FILE: "path/to/scenario.yaml",
Options.LOG_LEVEL: "info",
Options.OUTPUT: "output.pb",
Options.LOG_FILE: "scenario.log",
}
make_config(run_config)
You can also use the associated argument parser, to extract the run_config dynamically from a string:
from fameio.scripts.make_config import Options, run as make_config
from fameio.source.cli.make_config import handle_args
my_defaults = {Options.FILE: "path/to/scenario.yaml",
Options.LOG_LEVEL: "info",
Options.OUTPUT: "output.pb",
Options.LOG_FILE: "scenario.log",
}
my_arg_string = ['-f', 'my/other/scenario.yaml', '-l', 'error']
run_config = handle_args(my_arg_string, my_defaults)
make_config(run_config)
The "scenario.yaml" file contains all configuration options for a FAME-based simulation.
It consists of the sections Schema
, GeneralProperties
, Agents
and Contracts
, and the optional section StringSets
.
All of them are described below.
The Schema describes a model's components such as its types of agents, their inputs, what data they exchange, etc.
It is also used to validate the model inputs provided in the scenario.yaml
.
Since the Schema is valid until the model itself is changed, it is recommended to defined it in a separate file and include the file here.
Currently, the schema specifies:
The Schema consists of the sections JavaPackages
and AgentTypes
.
This section defines the name of the Java packages in which the model code is located.
A similar data set was formerly specified in the fameSetup.yaml
, but is now specified in the schema.
Each of the three sections Agents
, DataItems
, and Portables
contain a list of fully qualified java package names of your model's classes.
Package names can occur in multiple lists and may overlap.
It is not necessary (but possible) to specify the nearest enclosing package for each Agent, DataItem or Portable.
Specifying any super-package will also work.
Also, package names occur on multiple lists for Agent, DataItem or Portable.
For example, for a project with all its
the corresponding section in the schema would look like this:
JavaPackages:
Agents:
- "agents"
DataItems:
- "msg"
Portables:
- "portableItems"
- "otherPortables"
Here, each type of agent that can be created in your FAME-based application is listed, its attributes and its available Products for Contracts. The structure of this section
AgentTypes:
MyAgentType:
Attributes:
MyAttribute:
...
MyOtherAttribute:
...
Products: [ 'Product1', 'Product2', 'Product3' ]
MyOtherAgentWithoutProductsOrAttributes:
MyAgentType
Java's simple class name of the Agent typeAttributes
indicates that beginning of the attribute definition section for this Agent typeMyAttribute
Name of an attribute as specified in the corresponding Java source code of this Agent type (annotated with "@Input")MyOtherAttribute
Name of another attribute derived from Java source codeProducts
list of Products that this Agent can send in Contracts; derived from Java source code of this Agent type (annotated with "@Product")MyOtherAgentWithoutProductsOrAttributes
an Agent type that requires neither Attributes nor ProductsBoth Attributes and Products are optional - there could be useful Agents that require neither of them.
In the above example attribute definition was not shown (indicated by ...
).
The next example provides details on how to define an attribute:
MySimpleAttribute:
AttributeType: enum
Mandatory: true
List: false
Values: [ 'AllowedValue1', 'AllowedValue2' ]
Default: 'AllowedValue1'
Help: 'My help text'
MyComplexAttribute:
AttributeType: block
NestedAttributes:
InnerAttributeA:
AttributeType: integer
InnerAttributeB:
AttributeType: double
MySimpleAttribute
, MyDoubleList
, MyComplexAttribute
Names of the attributes as specified in the Java enum annotated with "@Input"AttributeType
(required) data type of the attribute; see options in table belowMandatory
(optional - true by default) if true: the attribute is required for this agent and validation will fail if the attribute is missing in the scenario and no default is providedList
(optional - false by default)
AttributeType: time_series
cannot be trueAttributeType: block
NestedAttributes
(required only if AttributeType: block
, otherwise disallowed) starts an inner Attribute definition block - defined Attributes are sub-elements of MyComplexAttribute
Values
(optional - None by default): if present defines a list of allowed values for this attributeDefault
(optional - None by default): if present defines a default value to be used in case the scenario does not specify itHelp
(optional - None by default): if present defines a help text to you attributeAttributeType | value |
---|---|
integer | a 32-bit integer value |
double | a 64-bit floating-point value (integers also allowed) |
long | a 64-bit integer value |
time_stamp | either a FAME time stamp string or 64-bit integer value |
string | any string |
string_set | a string from a set of allowed Values defined in StringSet section in scenario |
enum | a string from a set of allowed Values defined in schema |
time_series | either a path to a .csv-file or a single 64-bit floating-point value; does not support List: true |
block | this attribute has no value of its own but hosts a group of nested Attributes; implies NestedAttributes to be defined |
Specifies FAME-specific properties of the simulation. Structure:
GeneralProperties:
RunId: 1
Simulation:
StartTime: 2011-12-31_23:58:00
StopTime: 2012-12-30_23:58:00
RandomSeed: 1
Output:
Interval: 100
Process: 0
Parameters:
RunId
an ID that can be given to the simulation; use at your discretionStartTime
time stamp in the format YYYY-MM-DD_hh:mm:ss; first moment of the simulation.StopTime
time stamp in the format YYYY-MM-DD_hh:mm:ss; last moment of the simulation - i.e. simulation terminates
after passing that time stampRandomSeed
seed to initialise random number generation; each value leads to a unique series of random numbers.Parameters in section Output
are deprecated and will be removed in FAME-Io v3.0
Interval
number of simulation ticks in between write-to-disk events; may be used for performance optimisations;Process
id of process that performs write-to-disk operations; leave at 0 to be compatible with single-processes;Specifies all Agents to be created in the simulation in a list. Each Agent has its own entry. Structure:
Agents:
- Type: MyAgentWithInputs
Id: 1
Attributes:
MyEnum: SAME_SHARES
MyInteger: 2
MyDouble: 4.2
MyTimeSeries: "./path/to/time_series.csv"
- Type: MyAgentWithoutInputs
Id: 2
Agent Parameters:
Type
Mandatory; Java's simple class name of the agent to be createdId
Mandatory; simulation-unique id of this agent; if two agents have the same ID, the configuration process will
stop.Attributes
Optional; if the agent has any attributes, specify them here in the format "AttributeName: value"; please
see attribute table aboveThe specified Attributes
for each agent must match the specified Attributes
options in the linked Schema (see above).
For better structure and readability of the scenario.yaml
, Attributes
may also be specified in a nested way as demonstrated below.
Agents:
- Type: MyAgentWithInputs
Id: 1
Attributes:
Parent:
MyEnum: SAME_SHARES
MyInteger: 2
Parent2:
MyDouble: 4.2
Child:
MyTimeSeries: "./path/to/time_series.csv"
In case Attributes are defined with List: true
option, lists are assigned to an Attribute or Group:
Attributes:
MyDoubleList: [ 5.2, 4.5, 7, 9.9 ]
MyListGroup:
- IntValueA: 5
IntValueB: 42
- IntValueA: 7
IntValueB: 100
Here, MyDoubleList
and MyListGroup
need to specify List: true
in the corresponding Schema.
The shorter []
-notation was used to assign a list of floating-point values to MyDoubleList
.
Nested items IntValueA
and IntValueB
of MyListGroup
are assigned within a list, allowing the specification of these nested items several times.
Specifies all Contracts, i.e. repetitive bilateral transactions in between agents.
Contracts are given as a list.
We recommend moving Contracts to separate files and to use the !include
command to integrate them in the scenario.
Contracts:
- SenderId: 1
ReceiverId: 2
ProductName: ProductOfAgent_1
FirstDeliveryTime: -25
DeliveryIntervalInSteps: 3600
- SenderId: 2
ReceiverId: 1
ProductName: ProductOfAgent_2
FirstDeliveryTime: -22
DeliveryIntervalInSteps: 3600
Attributes:
ProductAppendix: value
TimeOffset: 42
Contract Parameters:
SenderId
unique ID of agent sending the productReceiverId
unique ID of agent receiving the productProductName
name of the product to be sentFirstDeliveryTime
first time of delivery in the format "seconds after the January 1st 2000, 00:00:00"DeliveryIntervalInSteps
delay time in between deliveries in secondsAttributes
can be set to include additional information as int
, float
, enum
, or dict
data typesOften, scenarios contain multiple agents of similar type that also have similar chains of contracts.
Therefore, FAME-Io supports a compact definition of multiple similar contracts.
SenderId
and ReceiverId
can both be lists and support One-to-N, N-to-One and N-to-N relations like in the following example:
Contracts:
# effectively 3 similar contracts (0 -> 11), (0 -> 12), (0 -> 13)
# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps
- SenderId: 0
ReceiverId: [ 11, 12, 13 ]
ProductName: MyOtherProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
# effectively 3 similar contracts (1 -> 10), (2 -> 10), (3 -> 10)
# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps
- SenderId: [ 1, 2, 3 ]
ReceiverId: 10
ProductName: MyProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
# effectively 3 similar contracts (1 -> 11), (2 -> 12), (3 -> 13)
# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps
- SenderId: [ 1, 2, 3 ]
ReceiverId: [ 11, 12, 13 ]
ProductName: MyThirdProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
Combined with YAML anchors complex contract chains can be easily reduced to a minimum of required configuration. The following example is equivalent to the previous one and allows a quick extension of contracts to a new couple of agents e.g. (4;14):
Groups:
- &agentList1: [ 1,2,3 ]
- &agentList2: [ 11,12,13 ]
Contracts:
- SenderId: 0
ReceiverId: *agentList2
ProductName: MyOtherProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
- SenderId: *agentList1
ReceiverId: 10
ProductName: MyProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
- SenderId: *agentList1
ReceiverId: *agentList2
ProductName: MyThirdProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
This optional section defines values of type string_set
.
In contrast to enum
values, which are statically defined in the Schema
, string_set
values can be dynamically defined in this section.
If an agent attribute is of type string_set
and the attribute is set in the scenario
, then
StringSets
in the scenario
must contain an entry named exactly like the attribute, andValues
declaration.For instance:
In schema
:
AgentTypes:
FuelsMarket:
Attributes:
FuelType:
AttributeType: string_set
In scenario
:
StringSets:
FuelType:
Values: ['OIL', 'HARD_COAL', 'LIGNITE']
Agents:
- Type: FuelsMarket
Id: 1
Attributes:
FuelType: OIL
Important: If different types of Agents shall refer to the same StringSet, their attributes in schema must have the exact same name.
TIME_SERIES inputs are not directly fed into the Scenario YAML file. Instead, TIME_SERIES reference a CSV file that can be stored some place else. These CSV files follow a specific structure:
YYYY-MM-DD_hh:mm:ss
#
You may add comments using #
.
Exemplary content of a valid CSV file:
# If you want an optional header, you must use a comment
2012-01-01_00:00:00;400
2013-01-01_00:00:00;720.5
2014-01-01_00:00:00;650
2015-01-01_00:00:00;99.27772
2016-01-01_00:00:00;42 # optional comment on this particular data point
2017-01-01_00:00:00;0.1
Please refer also to the detailed article about TimeStamps
in the FAME-Wiki.
The user may include other YAML files into a YAML file to divide the content across files as convenient.
We explicitly recommend using this feature for the Schema
and Contracts
sections.
Otherwise, the scenario.yaml may become crowded.
To hint YAML to load the content of another file use !include "path/relative/to/including/yaml/file.yml"
.
You can concatenate !include commands and can use !include in the included file as well.
The path to the included file is always relative to the file using the !include command.
So with the following file structure
a.yaml
folder/b.yaml
folder/c.yaml
folder/deeper_folder/d.yaml
the following !include commands work
ToBe: !include "folder/b.yaml"
OrNot: !include "folder/deeper_folder/d.yaml"
ThatIs: !include "c.yaml"
TheQuestion: !include "deeper_folder/d.yaml"
Provided that
Or: maybe
not: "?"
the resulting file would look like this:
ToBe:
ThatIs:
Or: maybe
TheQuestion:
not: "?"
OrNot:
not: "?"
You may also specify absolute file paths if preferred by starting with a "/".
When specifying only a file path, the complete content of the file is assigned to the given key. You always need a key to assign the !include command to. However, you cannot combine the value returned from !include with other values in the same key. Thus, the following combinations do not work:
!include "file.yaml" # no key assigned
Key:
Some: OtherItem
!include "file.yaml" # cannot join with other named items
List:
- an: entry
!include "file.yaml" # cannot directly join with list items, even if !include returns a list
Instead of including all content in the included file, you may also pick a specific node within that file.
For this use !include [<relative/path/to/file.yaml>, Path:To:Field:In:Yaml]
.
Here, :
is used in the node-specifying string to select a sequence of nodes to follow - with custom depth.
Consider the following two files:
Set1:
Subset1:
Key: Value
Set2:
OtherKey: OtherValue
- Type: MyAgentWithInputs
Id: 1
Attributes: !include_node [ file_to_be_included.yaml, Set1:Subset1 ]
Compiling "including_file.yaml" results in
- Type: MyAgentWithInputs
Id: 1
Attributes:
Key: Value
Using wildcards in the given path (e.g. "path/to/many/*.yaml") will lead to loading multiple files and assigning their content to the same key. You can make use of this feature with or without specifying a node selector. However, the elements to be joined across multiple files must be lists. These lists are then concatenated into a single list and then assigned to the key in the file calling !include. This feature is especially useful for Contracts: You can split the Contracts list into several files and place them in a separate folder. Then use !include to re-integrate them into your configuration. An example:
Contracts:
- ContractA
- ContractB
Contracts:
- ContractC
- ContractD
- ContractE
Contracts: [!include "my_contract*.yaml", "Contracts"]
results in
Contracts:
- ContractA
- ContractB
- ContractC
- ContractD
- ContractE
Files that have their name start with "IGNORE_" are not included with the !include command. You will see a debug output to notify you that the file was ignored. Use this to temporarily take files out ouf your configuration without deleting or moving them.
Takes an output file in protobuf format of FAME-based applications and converts it into files in CSV format. An individual file for each type of Agent is created in a folder named after the protobuf input file. Call structure:
convertFameResults -f <./path/to/protobuf_file.pb>
You may also specify any of the following arguments:
Command | Action |
---|---|
-l or --log | Sets the logging level. Default is WARNING . Options are DEBUG , INFO , WARNING , ERROR , CRITICAL . |
-lf or --logfile | Sets the logging file. Default is None . If None is provided, all logs get only printed to the console. |
-a or --agents | If specified, only a subset of agents is extracted from the protobuf file. Default is to extract all agents. |
-o or --output | Sets the path to where the generated output files are written to. If not specified, the folder's name is derived from the input file's name. Folder will be created if it does not exist. |
-se or --single-export | Enables export of individual agents to individual files, when present. If not present (the default) one file per AgentType is created. |
-m or --memory-saving | When specified, reduces memory usage profile at the cost of runtime. Use only when necessary. |
-cc or --complex-column | Defines how to deal with complex indexed output columns (if any). IGNORE ignores complex columns. SPLIT creates a separate file for each complex indexed output column. |
-t or --time | Option to define conversion of time steps to given format (default=UTC ) by -t/--time {UTC, INT, FAME} |
--input-recovery or --no-input-recovery | If True, all input data are recovered as well as the outputs (default=False). |
Additionally, you may merge TimeSteps
of a certain range of steps in the output files to
i) associate multiple time steps with a common logical time in your simulation
ii) reduce number of lines in output files
For this, add the option merge-times
and specify the arguments as follows:
Command | Action |
---|---|
-fp or --focal-point | TimeStep on which steps-before earlier and steps-after later TimeSteps are merged on |
-sb or --steps-before | Range of TimeSteps before the focal-point they get merged to |
-sa or --steps-after | Range of TimeSteps after the focal-point they get merged to |
This could look as follows:
convertFameResults -f <./path/to/protobuf_file.pb> -l debug -lf <path/to/output.log> -a AgentType1 AgentType2 -o myCsvFolder -m -cc SPLIT merge-times -fp 0 -sb 1799 -sa 1800
Make sure that in the range of time steps you specify for merging there is only one value per column in the merged time range. If multiple values per column are merged values will get concatenated and might yield unexpected results.
You may also call the conversion script from any Python script with:
from fameio.scripts.convert_results import Options, run as convert_results
convert_results({Options.FILE: "./path/to/protobuf_file.pb"})
Similar to the console call you may also specify custom run config arguments and add it in a dictionary to the function call.
from fameio.scripts.convert_results import Options, run as convert_results
run_config = {Options.FILE: "./path/to/protobuf_file.pb",
Options.LOG_LEVEL: "info",
Options.LOG_FILE: "scenario.log",
Options.OUTPUT: "Output",
Options.AGENT_LIST: ['AgentType1', 'AgentType2'],
Options.MEMORY_SAVING: False,
Options.SINGLE_AGENT_EXPORT: False,
Options.RESOLVE_COMPLEX_FIELD: "SPLIT",
Options.TIME: "INT",
Options.TIME_MERGING: {},
}
convert_results(run_config)
You can also use the associated argument parser, to extract the run_config dynamically from a string:
from fameio.scripts.convert_results import Options, run as convert_results
from fameio.source.cli.convert_results import handle_args
my_defaults = {Options.FILE: "./path/to/protobuf_file.pb",
Options.LOG_LEVEL: "info",
Options.LOG_FILE: "scenario.log",
Options.OUTPUT: "Output",
Options.AGENT_LIST: ['AgentType1', 'AgentType2'],
Options.MEMORY_SAVING: False,
Options.SINGLE_AGENT_EXPORT: False,
Options.RESOLVE_COMPLEX_FIELD: "SPLIT",
Options.TIME: "INT",
Options.TIME_MERGING: {},
}
my_arg_string = ['-f', 'my/other/scenario.yaml', '-l', 'error']
run_config = handle_args(my_arg_string, my_defaults)
convert_results(run_config)
If you use FAME-Io for academic work, please cite as follows.
Bibtex entry:
@article{fameio2023joss,
author = {Felix Nitsch and Christoph Schimeczek and Ulrich Frey and Benjamin Fuchs},
title = {FAME-Io: Configuration tools for complex agent-based simulations},
journal = {Journal of Open Source Software},
year = {2023},
doi = {doi: https://doi.org/10.21105/joss.04958}
}
This is a purely scientific project by (at the moment) one research group. Thus, there is no paid technical support available. However, we will give our best to answer your questions and provide support.
If you experience any trouble with FAME-Io, you may contact the developers via fame@dlr.de. Please report bugs and make feature requests by filing issues following the provided templates (see also Contribute). For substantial enhancements, we recommend that you contact us via fame@dlr.de for working together on the code in common projects or towards common publications and thus further develop FAME-Io.
FAQs
Python scripts for operation of FAME models
We found that fameio demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Product
We're launching a new set of license analysis and compliance features for analyzing, managing, and complying with licenses across a range of supported languages and ecosystems.
Product
We're excited to introduce Socket Optimize, a powerful CLI command to secure open source dependencies with tested, optimized package overrides.