goofi

goofi-pipe

Generative Organic Oscillation Feedback Isomorphism Pipeline

Installation

If you only want to run goofi-pipe and not edit any of the code, make sure you activated the desired Python environment with Python>=3.9 and run the following commands in your terminal:

pip install goofi # install goofi-pipe
goofi-pipe # start the application

[!NOTE] On some platforms (specifically Linux and Mac) it might be necessary to install the liblsl package for some of goofi-pipe's features (everything related to LSL streams). Follow the instructions provided here, or simply install it via

conda install -c conda-forge liblsl

Development

Follow these steps if you want to adapt the code of existing nodes, or create custom new nodes. In your terminal, make sure you activated the desired Python environment with Python>=3.9, and that you are in the directory where you want to install goofi-pipe. Then, run the following commands:

git clone [e-mail]:PhilippThoelke/goofi-pipe.git # download the repository
cd goofi-pipe # navigate into the repository
pip install -e . # install goofi-pipe in development mode
goofi-pipe # start the application to make sure the installation was successful

Basic Usage

Accessing the Node Menu

To access the node menu, simply double-click anywhere within the application window or press the 'Tab' key. The node menu allows you to add various functionalities to your pipeline. Nodes are categorized for easy access, but if you're looking for something specific, the search bar at the top is a handy tool.

Common Parameters and Metadata

Common Parameters: All nodes within goofi have a set of common parameters. These settings consistently dictate how the node operates within the pipeline.

• AutoTrigger: This option, when enabled, allows the node to be triggered automatically. When disabled, the node is triggered when it receives input.

• Max_Frequency: This denotes the maximum rate at which computations are set for the node.

Metadata: This section conveys essential information passed between nodes. Each output node will be accompanied by its metadata, providing clarity and consistency throughout the workflow.

Here are some conventional components present in the metadata

• Channel Dictionary: A conventional representation of EEG channels names.

• Sampling Frequency: The rate at which data samples are measured. It's crucial for maintaining consistent data input and output across various nodes.

• Shape of the Output: Details the format and structure of the node's output.

Playing with Pre-recorded EEG Signal using LslStream

This image showcases the process of utilizing a pre-recorded EEG signal through the LslStream node. It's crucial to ensure that the Stream Name in the LslStream node matches the stream name in the node receiving the data. This ensures data integrity and accurate signal processing in real-time.

Patch examples

Basic Signal Processing Patch

This patch provides a demonstration of basic EEG signal processing using goofi-pipe.

1. EegRecording: This is the starting point where the EEG data originates.

2. LslClient: The LslClient node retrieves the EEG data from EegRecording. Here, the visual representation of the EEG data being streamed in real-time is depicted. By default, the multiple lines in the plot correspond to the different EEG channels.

3. Buffer: This node holds the buffered EEG data.

4. Psd: Power Spectral Density (PSD) is a technique to measure a signal's power content versus frequency. In this node, the raw EEG data is transformed to exhibit its power distribution across distinct frequency bands.

5. Math: This node is employed to execute mathematical operations on the data. In this context, it's rescaling the values to ensure a harmonious dynamic range between 0 and 1, which is ideal for image representation. The resultant data is then visualized as an image.

One of the user-friendly features of goofi-pipe is the capability to toggle between different visualizations. By 'Ctrl+clicking' on any plot within a node, you can effortlessly switch between a line plot and an image representation, offering flexibility in data analysis.

Sending Power Bands via Open Sound Control (OSC)

Expanding on the basic patch, the advanced additions include:

• Select: Chooses specific EEG channels.
• PowerBandEEG: Computes EEG signal power across various frequency bands.
• ExtendedTable: Prepares data for transmission in a structured format.
• OscOut: Sends data using the Open-Sound-Control (OSC) protocol.

These nodes elevate data processing and communication capabilities.

Real-Time Connectivity and Spectrogram

This patch highlights:

• Connectivity: Analyzes relationships between EEG channels, offering selectable methods like wPLI, coherence, PLI, and more.

• Spectrogram: Created using the PSD node followed by a Buffer, it provides a time-resolved view of the EEG signal's frequency content.

Principal Component Analysis (PCA)

Using PCA (Principal Component Analysis) allows us to reduce the dimensionality of raw EEG data, while retaining most of the variance. We use the first three components and visualize their trajectory, allowing us to identify patterns in the data over time. The topographical maps show the contrbution of each channel to the first four principal components (PCs).

Realtime Classification

leverage the multimodal framework of goofi, state-of-the-art machine learning classifiers can be built on-the-fly to predict behavior from an array of different sources. Here's a brief walkthrough of three distinct examples:

1. EEG Signal Classification

This patch captures raw EEG signals using the EEGrecording and LslStreammodule. The classifier module allows to capture data from different states indicated by the user from n features, which in the present case are the 64 EEG channels. Some classifiers allow for visualization of feature importance. Here we show a topomap of the distribution of features importances on the scalp. The classifier outputs probability of being in each of the states in the training data. This prediction is smoothed using a buffer for less jiterry results.

2. Audio Input Classification

The audio input stream captures real-time sound data, which can also be passed through a classifier. Different sonic states can be predicted in realtime.

3. Video Input Classification

In this example, video frames are extracted using the VideoStream module. Similarly, prediction of labelled visual states can be achieved in realtime. The images show how two states (being on the left or the right side of the image) can be detected using classification

These patches demonstrate the versatility of our framework in handling various types of real-time data streams for classification tasks.

Musical Features using Biotuner

This patch presents a pipeline for processing EEG data to extract musical features:

• Data flows from the EEG recording through several preprocessing nodes and culminates in the Biotuner node, which specializes in deriving musical attributes from the EEG.

• Biotuner Node: With its sophisticated algorithms, Biotuner pinpoints harmonic relationships, tension, peaks, and more, essential for music theory analysis.

Delving into the parameters of the Biotuner node:

• N Peaks: The number of spectral peaks to consider.
• F Min & F Max: Defines the frequency range for analysis.
• Precision: Sets the precision in Hz for peak extraction.
• Peaks Function: Method to compute the peaks, like EMD, fixed band, or harmonic recurrence.
• N Harm Subharm & N Harm Extended: Configures number of harmonics used in different computations.
• Delta Lim: Defines the maximal distance between two subharmonics to include in subharmonic tension computation.

For a deeper understanding and advanced configurations, consult the Biotuner repository.

Data Types

To simplify understanding, we've associated specific shapes with data types at the inputs and outputs of nodes:

• Circles: Represent arrays.
• Triangles: Represent strings.
• Squares: Represent tables.

Node Categories

Analysis

Nodes that perform analysis on the data.

View Nodes

 AudioTagging

• Inputs:
  • audioIn: ARRAY
• Outputs:
  • tags: STRING
  • probabilities: ARRAY
  • embedding: ARRAY

 Avalanches

• Inputs:
  • data: ARRAY
• Outputs:
  • size: ARRAY
  • duration: ARRAY

 Binarize

• Inputs:
  • data: ARRAY
• Outputs:
  • bin_data: ARRAY

 Bioelements

• Inputs:
  • data: ARRAY
• Outputs:
  • elements: TABLE

 Bioplanets

• Inputs:
  • peaks: ARRAY
• Outputs:
  • planets: TABLE
  • top_planets: STRING

 Biorhythms

• Inputs:
  • tuning: ARRAY
• Outputs:
  • pulses: ARRAY
  • steps: ARRAY
  • offsets: ARRAY

 Biotuner

• Inputs:
  • data: ARRAY
• Outputs:
  • harmsim: ARRAY
  • tenney: ARRAY
  • subharm_tension: ARRAY
  • cons: ARRAY
  • peaks_ratios_tuning: ARRAY
  • harm_tuning: ARRAY
  • peaks: ARRAY
  • amps: ARRAY
  • extended_peaks: ARRAY
  • extended_amps: ARRAY

 CardiacRespiration

• Inputs:
  • data: ARRAY
• Outputs:
  • cardiac: ARRAY

 CardioRespiratoryVariability

• Inputs:
  • data: ARRAY
• Outputs:
  • MeanNN: ARRAY
  • SDNN: ARRAY
  • SDSD: ARRAY
  • RMSSD: ARRAY
  • pNN50: ARRAY
  • LF: ARRAY
  • HF: ARRAY
  • LF/HF: ARRAY
  • LZC: ARRAY

 Classifier

• Inputs:
  • data: ARRAY
• Outputs:
  • probs: ARRAY
  • feature_importances: ARRAY

 Clustering

• Inputs:
  • matrix: ARRAY
• Outputs:
  • cluster_labels: ARRAY
  • cluster_centers: ARRAY

 Compass

• Inputs:
  • north: ARRAY
  • south: ARRAY
  • east: ARRAY
  • west: ARRAY
• Outputs:
  • angle: ARRAY

 Connectivity

• Inputs:
  • data: ARRAY
• Outputs:
  • matrix: ARRAY

 Coord2loc

• Inputs:
  • latitude: ARRAY
  • longitude: ARRAY
• Outputs:
  • coord_info: TABLE

 Correlation

• Inputs:
  • data1: ARRAY
  • data2: ARRAY
• Outputs:
  • pearson: ARRAY

 DissonanceCurve

• Inputs:
  • peaks: ARRAY
  • amps: ARRAY
• Outputs:
  • dissonance_curve: ARRAY
  • tuning: ARRAY
  • avg_dissonance: ARRAY

 EigenDecomposition

• Inputs:
  • matrix: ARRAY
• Outputs:
  • eigenvalues: ARRAY
  • eigenvectors: ARRAY

 ERP

• Inputs:
  • signal: ARRAY
  • trigger: ARRAY
• Outputs:
  • erp: ARRAY

 FacialExpression

• Inputs:
  • image: ARRAY
• Outputs:
  • emotion_probabilities: ARRAY
  • action_units: ARRAY
  • main_emotion: STRING

 Fractality

• Inputs:
  • data_input: ARRAY
• Outputs:
  • fractal_dimension: ARRAY

 GraphMetrics

• Inputs:
  • matrix: ARRAY
• Outputs:
  • clustering_coefficient: ARRAY
  • characteristic_path_length: ARRAY
  • betweenness_centrality: ARRAY
  • degree_centrality: ARRAY
  • assortativity: ARRAY
  • transitivity: ARRAY

 HarmonicSpectrum

• Inputs:
  • psd: ARRAY
• Outputs:
  • harmonic_spectrum: ARRAY
  • max_harmonicity: ARRAY
  • avg_harmonicity: ARRAY

 Img2Txt

• Inputs:
  • image: ARRAY
• Outputs:
  • generated_text: STRING

 LempelZiv

• Inputs:
  • data: ARRAY
• Outputs:
  • lzc: ARRAY

 PCA

• Inputs:
  • data: ARRAY
• Outputs:
  • principal_components: ARRAY

 PoseEstimation

• Inputs:
  • image: ARRAY
• Outputs:
  • pose: ARRAY

 PowerBand

• Inputs:
  • data: ARRAY
• Outputs:
  • power: ARRAY

 PowerBandEEG

• Inputs:
  • data: ARRAY
• Outputs:
  • delta: ARRAY
  • theta: ARRAY
  • alpha: ARRAY
  • lowbeta: ARRAY
  • highbeta: ARRAY
  • gamma: ARRAY

 ProbabilityMatrix

• Inputs:
  • input_data: ARRAY
• Outputs:
  • data: ARRAY

 SpectroMorphology

• Inputs:
  • data: ARRAY
• Outputs:
  • spectro: ARRAY

 SpeechSynthesis

• Inputs:
  • text: STRING
  • voice: ARRAY
• Outputs:
  • speech: ARRAY
  • transcript: STRING

 TransitionalHarmony

• Inputs:
  • data: ARRAY
• Outputs:
  • trans_harm: ARRAY
  • melody: ARRAY

 TuningColors

• Inputs:
  • data: ARRAY
• Outputs:
  • hue: ARRAY
  • saturation: ARRAY
  • value: ARRAY
  • color_names: STRING

 TuningMatrix

• Inputs:
  • tuning: ARRAY
• Outputs:
  • matrix: ARRAY
  • metric_per_step: ARRAY
  • metric: ARRAY

 TuningReduction

• Inputs:
  • tuning: ARRAY
• Outputs:
  • reduced: ARRAY

 VAMP

• Inputs:
  • data: ARRAY
• Outputs:
  • comps: ARRAY

 VocalExpression

• Inputs:
  • data: ARRAY
• Outputs:
  • prosody_label: STRING
  • burst_label: STRING
  • prosody_score: ARRAY
  • burst_score: ARRAY

Array

Nodes implementing array operations.

View Nodes

 Clip

• Inputs:
  • array: ARRAY
• Outputs:
  • out: ARRAY

 Join

• Inputs:
  • a: ARRAY
  • b: ARRAY
• Outputs:
  • out: ARRAY

 Math

• Inputs:
  • data: ARRAY
• Outputs:
  • out: ARRAY

 Operation

• Inputs:
  • a: ARRAY
  • b: ARRAY
• Outputs:
  • out: ARRAY

 Reduce

• Inputs:
  • array: ARRAY
• Outputs:
  • out: ARRAY

 Reshape

• Inputs:
  • array: ARRAY
• Outputs:
  • out: ARRAY

 Select

• Inputs:
  • data: ARRAY
• Outputs:
  • out: ARRAY

 Transpose

• Inputs:
  • array: ARRAY
• Outputs:
  • out: ARRAY

Inputs

Nodes that provide data to the pipeline.

View Nodes

 Audiocraft

• Inputs:
  • prompt: STRING
• Outputs:
  • wav: ARRAY

 AudioStream

• Inputs:
• Outputs:
  • out: ARRAY

 ConstantArray

• Inputs:
• Outputs:
  • out: ARRAY

 ConstantString

• Inputs:
• Outputs:
  • out: STRING

 EEGRecording

• Inputs:
• Outputs:

 ExtendedTable

• Inputs:
  • base: TABLE
  • array_input1: ARRAY
  • array_input2: ARRAY
  • array_input3: ARRAY
  • array_input4: ARRAY
  • array_input5: ARRAY
  • string_input1: STRING
  • string_input2: STRING
  • string_input3: STRING
  • string_input4: STRING
  • string_input5: STRING
• Outputs:
  • table: TABLE

 FractalImage

• Inputs:
  • complexity: ARRAY
• Outputs:
  • image: ARRAY

 ImageGeneration

• Inputs:
  • prompt: STRING
  • negative_prompt: STRING
  • base_image: ARRAY
• Outputs:
  • img: ARRAY

 Kuramoto

• Inputs:
  • initial_phases: ARRAY
• Outputs:
  • phases: ARRAY
  • coupling: ARRAY
  • order_parameter: ARRAY
  • waveforms: ARRAY

 LoadFile

• Inputs:
• Outputs:
  • data_output: ARRAY

 LSLClient

• Inputs:
• Outputs:
  • out: ARRAY

 MeteoMedia

• Inputs:
  • latitude: ARRAY
  • longitude: ARRAY
  • location_name: STRING
• Outputs:
  • weather_data_table: TABLE

 OSCIn

• Inputs:
• Outputs:
  • message: TABLE

 PromptBook

• Inputs:
  • input_prompt: STRING
• Outputs:
  • out: STRING

 Reservoir

• Inputs:
  • connectivity: ARRAY
• Outputs:
  • data: ARRAY

 SerialStream

• Inputs:
• Outputs:
  • out: ARRAY

 Sine

• Inputs:
• Outputs:
  • out: ARRAY

 Table

• Inputs:
  • base: TABLE
  • new_entry: ARRAY
• Outputs:
  • table: TABLE

 TextGeneration

• Inputs:
  • prompt: STRING
• Outputs:
  • generated_text: STRING

 VideoStream

• Inputs:
• Outputs:
  • frame: ARRAY

 ZeroMQIn

• Inputs:
• Outputs:
  • data: ARRAY

Misc

Miscellaneous nodes that do not fit into other categories.

View Nodes

 AppendTables

• Inputs:
  • table1: TABLE
  • table2: TABLE
• Outputs:
  • output_table: TABLE

 ColorEnhancer

• Inputs:
  • image: ARRAY
• Outputs:
  • enhanced_image: ARRAY

 EdgeDetector

• Inputs:
  • image: ARRAY
• Outputs:
  • edges: ARRAY

 FormatString

• Inputs:
  • input_string_1: STRING
  • input_string_2: STRING
  • input_string_3: STRING
  • input_string_4: STRING
  • input_string_5: STRING
  • input_string_6: STRING
  • input_string_7: STRING
  • input_string_8: STRING
  • input_string_9: STRING
  • input_string_10: STRING
• Outputs:
  • output_string: STRING

 HSVtoRGB

• Inputs:
  • hsv_image: ARRAY
• Outputs:
  • rgb_image: ARRAY

 JoinString

• Inputs:
  • string1: STRING
  • string2: STRING
  • string3: STRING
  • string4: STRING
  • string5: STRING
• Outputs:
  • output: STRING

 RGBtoHSV

• Inputs:
  • rgb_image: ARRAY
• Outputs:
  • hsv_image: ARRAY

 SetMeta

• Inputs:
  • array: ARRAY
• Outputs:
  • out: ARRAY

 StringAwait

• Inputs:
  • message: STRING
  • trigger: ARRAY
• Outputs:
  • out: STRING

 TableSelectArray

• Inputs:
  • input_table: TABLE
• Outputs:
  • output_array: ARRAY

 TableSelectString

• Inputs:
  • input_table: TABLE
• Outputs:
  • output_string: STRING

Outputs

Nodes that send data to external systems.

View Nodes

 AudioOut

• Inputs:
  • data: ARRAY
• Outputs:
  • finished: ARRAY

 LSLOut

• Inputs:
  • data: ARRAY
• Outputs:

 MidiCCout

• Inputs:
  • cc1: ARRAY
  • cc2: ARRAY
  • cc3: ARRAY
  • cc4: ARRAY
  • cc5: ARRAY
• Outputs:
  • midi_status: STRING

 MidiOut

• Inputs:
  • note: ARRAY
  • velocity: ARRAY
  • duration: ARRAY
• Outputs:
  • midi_status: STRING

 OSCOut

• Inputs:
  • data: TABLE
• Outputs:

 SharedMemOut

• Inputs:
  • data: ARRAY
• Outputs:

 WriteCsv

• Inputs:
  • table_input: TABLE
• Outputs:

 ZeroMQOut

• Inputs:
  • data: ARRAY
• Outputs:

Signal

Nodes implementing signal processing operations.

View Nodes

 Buffer

• Inputs:
  • val: ARRAY
• Outputs:
  • out: ARRAY

 Cycle

• Inputs:
  • signal: ARRAY
• Outputs:
  • cycle: ARRAY

 EMD

• Inputs:
  • data: ARRAY
• Outputs:
  • IMFs: ARRAY

 FFT

• Inputs:
  • data: ARRAY
• Outputs:
  • mag: ARRAY
  • phase: ARRAY

 Filter

• Inputs:
  • data: ARRAY
• Outputs:
  • filtered_data: ARRAY

 FOOOFaperiodic

• Inputs:
  • psd_data: ARRAY
• Outputs:
  • offset: ARRAY
  • exponent: ARRAY
  • cf_peaks: ARRAY
  • cleaned_psd: ARRAY

 FrequencyShift

• Inputs:
  • data: ARRAY
• Outputs:
  • out: ARRAY

 Hilbert

• Inputs:
  • data: ARRAY
• Outputs:
  • inst_amplitude: ARRAY
  • inst_phase: ARRAY
  • inst_frequency: ARRAY

 IFFT

• Inputs:
  • spectrum: ARRAY
  • phase: ARRAY
• Outputs:
  • reconstructed: ARRAY

 PSD

• Inputs:
  • data: ARRAY
• Outputs:
  • psd: ARRAY

 Recurrence

• Inputs:
  • input_array: ARRAY
• Outputs:
  • recurrence_matrix: ARRAY
  • RR: ARRAY
  • DET: ARRAY
  • LAM: ARRAY

 Resample

• Inputs:
  • data: ARRAY
• Outputs:
  • out: ARRAY

 ResampleJoint

• Inputs:
  • data1: ARRAY
  • data2: ARRAY
• Outputs:
  • out1: ARRAY
  • out2: ARRAY

 Smooth

• Inputs:
  • data: ARRAY
• Outputs:
  • out: ARRAY

 StaticBaseline

• Inputs:
  • data: ARRAY
• Outputs:
  • normalized: ARRAY

 Threshold

• Inputs:
  • data: ARRAY
• Outputs:
  • thresholded: ARRAY

 TimeDelayEmbedding

• Inputs:
  • input_array: ARRAY
• Outputs:
  • embedded_array: ARRAY

 WelfordsZTransform

• Inputs:
  • data: ARRAY
• Outputs:
  • normalized: ARRAY