Multi-Agent Accelerator for Data Science Using Transactional Machine Learning (MAADSTML)
Revolutionizing Data Stream Science with Transactional Machine Learning
Overview
MAADSTML combines Artificial Intelligence, ChatGPT, PrivateGPT, Auto Machine Learning with Data Streams Integrated with Apache Kafka (or Redpanda) to create frictionless and elastic machine learning solutions.
This library allows users to harness the power of agent-based computing using hundreds of advanced linear and non-linear algorithms. Users can easily integrate Predictive Analytics, Prescriptive Analytics, Pre-Processing, and Optimization in any data stream solution by wrapping additional code around the functions below. It connects with Apache KAFKA brokers for cloud based computing using Kafka (or Redpanda) as the data backbone.
If analysing MILLIONS of IoT devices, you can easily deploy thousands of VIPER/HPDE instances in Kubernetes Cluster in AWS/GCP/Azure.
It uses VIPER as a KAFKA connector and seamlessly combines Auto Machine Learning, with Real-Time Machine Learning, Real-Time Optimization and Real-Time Predictions while publishing these insights in to a Kafka cluster in real-time at scale, while allowing users to consume these insights from anywhere, anytime and in any format.
It also HPDE as the AutoML technology for TML. Linux/Windows/Mac versions can be downloaded from Github
It uses VIPERviz to visualize streaming insights over HTTP(S). Linux/Windows/Mac versions can be downloaded from Github
MAADSTML details can be found in the book: Transactional Machine Learning with Data Streams and AutoML
To install this library a request should be made to support@otics.ca for a username and a MAADSTOKEN. Once you have these credentials then install this Python library.
Compatibility - Python 3.8 or greater - Minimal Python skills needed
Copyright
Installation
MAADS-VIPER Connector to Manage Apache KAFKA:
**TML is integrated with PrivateGPT (https://github.com/imartinez/privateGPT), which is a production ready GPT, that is 100% Local, 100% Secure and 100% FREE GPT Access.
Users need to PULL and RUN one of the privateGPT Docker containers:
1. Docker Hub: maadsdocker/tml-privategpt-no-gpu-amd64 (without NVIDIA GPU for AMD64 Chip)
2. Docker Hub: maadsdocker/tml-privategpt-with-gpu-amd64 (with NVIDIA GPU for AMD64 Chip)
3. Docker Hub: maadsdocker/tml-privategpt-no-gpu-arm64 (without NVIDIA GPU for ARM64 Chip)
4. Docker Hub: maadsdocker/tml-privategpt-with-gpu-arm64 (with NVIDIA GPU for ARM64 Chip)
Additional details are here: https://github.com/smaurice101/raspberrypi/tree/main/privategpt
TML accesses privateGPT container using REST API.
For PrivateGPT production deployments it is recommended that machines have the NVIDIA GPU as this will lead to significant performance improvements.
pgptingestdocs
pgptgetingestedembeddings
pgptchat
pgptdeleteembeddings
pgpthealth
vipermirrorbrokers
viperstreamquery
viperstreamquerybatch
viperlisttopics
viperdeactivatetopic
viperactivatetopic
vipercreatetopic
viperstats
vipersubscribeconsumer
viperunsubscribeconsumer
viperhpdetraining
viperhpdetrainingbatch
viperhpdepredict
viperhpdepredictprocess
viperhpdepredictbatch
viperhpdeoptimize
viperhpdeoptimizebatch
viperproducetotopic
viperproducetotopicbulk
viperconsumefromtopic
viperconsumefromtopicbatch
viperconsumefromstreamtopic
vipercreateconsumergroup
viperconsumergroupconsumefromtopic
viperproducetotopicstream
viperpreprocessrtms
viperpreprocessproducetotopicstream
Users can pre-process data streams using the following functions: MIN, MAX, AVG, COUNT, COUNTSTR, DIFF, DIFFMARGIN, SUM, MEDIAN, VARIANCE, OUTLIERS, OUTLIERSX-Y,VARIED, ANOMPROB,ANOMPROBX-Y,ENTROPY, AUTOCORR, TREND, CONSISTENCY, IQR (InterQuartileRange), Midhinge, GM (Geometric mean), HM (Harmonic mean), Trimean, CV (coefficient of Variation),Mad (Mean absolute deviation), Skewness, Kurtosis, Spikedetect, Unique, Uniquestr, Timediff: time should be in this layout:2006-01-02T15:04:05, Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the average time in seconds between consecutive dates.. Spikedetect uses a Zscore method to detect spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5. Geodiff (returns distance in Kilometers between two lat/long points)
Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from current local time. You can specify the UTC offset to adjust the current time to match the timezone of the data stream. You can specify timetype as millisecond, second, minute, hour, day. For example, if Dataage_1_minute, then this processtype will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference between the data stream timestamp and current local time and return the difference in minutes. This is a very powerful processtype for data quality and data assurance programs for any number of data streams.
Unique Checks numeric data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Uniquestr Checks string data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Uniquecount Checks numeric data for duplication. Returns count of unique numbers.
Uniquestrcount Checks string data for duplication. Returns count of unique strings.
CONSISTENCY checks if the data all have consistent data types. Returns 1 for consistent data types, 0 otherwise.
Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high
RAW for no processing.
ANOMPROB=Anomaly Probability, it will run several algorithms on the data stream window to determine a probability percentage of anomalous behaviour. This can be cross-referenced with other process types. This is very useful if you want to extract aggregate values that you can then use to build TML models and/or make decisions to prevent issues. ENTROPY will compute the amount of information in the data stream. AUTOCORR will run a autocorrelation regression: Y = Y (t-1), to indicate how previous value correlates with future value. TREND will run a linear regression of Y = f(Time), to determine if the data in the stream are increasing or decreasing.
ANOMPROBX-Y (similar to OUTLIERSX-Y), where X and Y are numbers or "n", if "n" means examine all anomalies for recurring patterns. They allow you to check if the anomalies in the streams are truly anomalies and not some pattern. For example, if a IoT device shuts off and turns on again routinely, this may be picked up as an anomaly when in fact it is normal behaviour. So, to ignore these cases, if ANOMPROB2-5, this tells Viper, check anomalies with patterns of 2-5 peaks. If the stream has two classes and these two classes are like 0 and 1000, and show a pattern, then they should not be considered an anomaly. Meaning, class=0, is the device shutting down, class=1000 is the device turning back on. If ANOMPROB3-10, Viper will check for patterns of classes 3 to 10 to see if they recur routinely. This is very helpful to reduce false positives and false negatives.
viperpreprocessbatch
vipercreatejointopicstreams
vipercreatetrainingdata
vipermodifyconsumerdetails
vipermodifytopicdetails
vipergroupdeactivate
vipergroupactivate
viperdeletetopics
viperanomalytrain
Perform anomaly/peer group analysis on text or numeric data stream using advanced unsupervised learning. VIPER automatically joins streams, and determines the peer group of "usual" behaviours using proprietary algorithms, which are then used to predict anomalies with viperanomalypredict in real-time. Users can use several parameters to fine tune the peer groups.
VIPER is one of the very few, if not only, technology to do anomaly/peer group analysis using unsupervised learning on data streams with Apache Kafka.
viperanomalytrainbatch
viperanomalypredict
VIPER is one of the very few, if not only, technology to do anomaly detection/predictions using unsupervised learning on data streams with Apache Kafka.
viperanomalypredictbatch
viperstreamcorr
viperpreprocesscustomjson
viperstreamcluster
vipersearchanomaly
vipernlp
viperchatgpt
viperexractpdffields
viperexractpdffieldbylabel
videochatloadresponse
areyoubusy
First import the Python library.
import maadstml
1. maadstml.viperstats(vipertoken,host,port=-999,brokerhost='',brokerport=-999,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: A JSON formatted object of all the Kafka broker information.
2. maadstml.vipersubscribeconsumer(vipertoken,host,port,topic,companyname,contactname,contactemail, location,description,brokerhost='',brokerport=-999,groupid='',microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
companyname : string, required
contactname : string, required
contactemail : string, required
location : string, required
description : string, required
brokerhost : string, optional
brokerport : int, optional
groupid : string, optional
microserviceid : string, optional
RETURNS: Consumer ID that the user must use to receive insights from topic.
3. maadstml.viperunsubscribeconsumer(vipertoken,host,port,consumerid,brokerhost='',brokerport=-999, microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
consumerid : string, required
brokerhost : string, optional
brokerport : int, optional
RETURNS: Success/failure
4. maadstml.viperproducetotopic(vipertoken,host,port,topic,producerid,enabletls=0,delay=100,inputdata='',maadsalgokey='', maadstoken='',getoptimal=0,externalprediction='',subtopics='',topicid=-999,identifier='',array=0,brokerhost='', brokerport=-999,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
subtopic : string, optional
topicid : int, optional
"array* : int, optional
identifier : string, optional
producerid : string, required
enabletls : int, optional
delay: int, optional
inputdata : string, optional
maadsalgokey : string, optional
maadstoken : string, optional
getoptimal: int, optional
externalprediction : string, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: Returns the value produced or results retrieved from the optimization.
4.1. maadstml.viperproducetotopicbulk(vipertoken,host,port,topic,producerid,inputdata,partitionsize=100,enabletls=1,delay=100, brokerhost='',brokerport=-999,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
producerid : string, required
inputdata : string, required
partitionsize : int, optional
enabletls : int, optional
delay: int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: None
5. maadstml.viperconsumefromtopic(vipertoken,host,port,topic,consumerid,companyname,partition=-1,enabletls=0,delay=100,offset=0, brokerhost='',brokerport=-999,microserviceid='',topicid='-999',rollbackoffsets=0,preprocesstype='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
preprocesstype : string, optional
If you only want to search for record that have a particular processtype, you can enter: MIN, MAX, AVG, COUNT, COUNTSTR, DIFF, DIFFMARGIN, SUM, MEDIAN, VARIANCE, OUTLIERS, OUTLIERSX-Y, VARIED, ANOMPROB,ANOMPROBX-Y,ENTROPY, AUTOCORR, TREND, CONSISTENCY, Unique, Uniquestr, Geodiff (returns distance in Kilometers between two lat/long points) IQR (InterQuartileRange), Midhinge, GM (Geometric mean), HM (Harmonic mean), Trimean, CV (coefficient of Variation), Mad (Mean absolute deviation), Skewness, Kurtosis, Spikedetect, Timediff: time should be in this layout:2006-01-02T15:04:05, Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the average time in seconds between consecutive dates. Spikedetect uses a Zscore method to detect spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5.
Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from current local time. You can specify the UTC offset to adjust the current time to match the timezone of the data stream. You can specify timetype as millisecond, second, minute, hour, day. For example, if Dataage_1_minute, then this processtype will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference between the data stream timestamp and current local time and return the difference in minutes. This is a very powerful processtype for data quality and data assurance programs for any number of data streams.
Unique Checks numeric data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Uniquestr Checks string data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Uniquecount Checks numeric data for duplication. Returns count of unique numbers.
Uniquestrcount Checks string data for duplication. Returns count of unique strings.
CONSISTENCY checks if the data all have consistent data types. Returns 1 for consistent data types, 0 otherwise.
Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high
RAW for no processing.
ANOMPROB=Anomaly probability, it will run several algorithms on the data stream window to determine a probaility of anomalous behaviour. This can be cross-refenced with OUTLIERS. It can be very powerful way to detection issues with devices.
ANOMPROBX-Y (similar to OUTLIERSX-Y), where X and Y are numbers, or "n". If "n", means examine all anomalies for patterns. They allow you to check if the anomalies in the streams are truly anomalies and not some pattern. For example, if a IoT device shuts off and turns on again routinely, this may be picked up as an anomaly when in fact it is normal behaviour. So, to ignore these cases, if ANOMPROB2-5, this tells Viper, check anomalies with patterns of 2-5 peaks. If the stream has two classes and these two classes are like 0 and 1000, and show a pattern, then they should not be considered an anomaly. Meaning, class=0, is the device shutting down, class=1000 is the device turning back on. If ANOMPROB3-10, Viper will check for patterns of classes 3 to 10 to see if they recur routinely. This is very helpful to reduce false positives and false negatives.
topicid : string, optional
rollbackoffsets : int, optional, enter value between 0 and 100
consumerid : string, required
companyname : string, required
partition : int, optional
enabletls: int, optional
delay: int, optional
offset: int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the contents read from the topic.
5.1 maadstml.viperconsumefromtopicbatch(vipertoken,host,port,topic,consumerid,companyname,partition=-1,enabletls=0,delay=100,offset=0, brokerhost='',brokerport=-999,microserviceid='',topicid='-999',rollbackoffsets=0,preprocesstype='',timedelay=0,asynctimeout=120)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
asynctimeout : int, optional
-This is the timeout in seconds for the Python library async function.
timedelay : int, optional
topic : string, required
preprocesstype : string, optional
If you only want to search for record that have a particular processtype, you can enter:
MIN, MAX, AVG, COUNT, COUNTSTR, DIFF, DIFFMARGIN, SUM, MEDIAN, VARIANCE, OUTLIERS, OUTLIERSX-Y, VARIED, ANOMPROB,ANOMPROBX-Y,ENTROPY, AUTOCORR, TREND,
IQR (InterQuartileRange), Midhinge, CONSISTENCY, GM (Geometric mean), HM (Harmonic mean), Trimean, CV (coefficient of Variation),
Mad (Mean absolute deviation), Skewness, Kurtosis, Spikedetect, Unique, Uniquestr, Timediff: time should be in this layout:2006-01-02T15:04:05,
Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the
average time in seconds between consecutive dates.
Spikedetect uses a Zscore method to detect spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5.
Geodiff (returns distance in Kilometers between two lat/long points)
Unique Checks numeric data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from current local time. You can specify the UTC offset to adjust the current time to match the timezone of the data stream. You can specify timetype as millisecond, second, minute, hour, day. For example, if Dataage_1_minute, then this processtype will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference between the data stream timestamp and current local time and return the difference in minutes. This is a very powerful processtype for data quality and data assurance programs for any number of data streams.
Uniquestr Checks string data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Uniquecount Checks numeric data for duplication. Returns count of unique numbers.
Uniquestrcount Checks string data for duplication. Returns count of unique strings.
CONSISTENCY checks if the data all have consistent data types. Returns 1 for consistent data types, 0 otherwise.
Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high
RAW for no processing.
ANOMPROB=Anomaly probability, it will run several algorithms on the data stream window to determine a probaility of anomalous behaviour. This can be cross-refenced with OUTLIERS. It can be very powerful way to detection issues with devices.
ANOMPROBX-Y (similar to OUTLIERSX-Y), where X and Y are numbers, or "n". If "n", means examine all anomalies for patterns. They allow you to check if the anomalies in the streams are truly anomalies and not some pattern. For example, if a IoT device shuts off and turns on again routinely, this may be picked up as an anomaly when in fact it is normal behaviour. So, to ignore these cases, if ANOMPROB2-5, this tells Viper, check anomalies with patterns of 2-5 peaks. If the stream has two classes and these two classes are like 0 and 1000, and show a pattern, then they should not be considered an anomaly. Meaning, class=0, is the device shutting down, class=1000 is the device turning back on. If ANOMPROB3-10, Viper will check for patterns of classes 3 to 10 to see if they recur routinely. This is very helpful to reduce false positives and false negatives.
topicid : string, required
rollbackoffsets : int, optional, enter value between 0 and 100
consumerid : string, required
companyname : string, required
partition : int, optional
enabletls: int, optional
delay: int, optional
offset: int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the contents read from the topic.
6. maadstml.viperhpdepredict(vipertoken,host,port,consumefrom,produceto,companyname,consumerid,producerid, hpdehost,inputdata,maxrows=0,algokey='',partition=-1,offset=-1,enabletls=1,delay=1000,hpdeport=-999,brokerhost='', brokerport=-999,timeout=120,usedeploy=0,microserviceid='',topicid=-999, maintopic='', streamstojoin='', array=0,pathtoalgos='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topicid : int, optional
pathtoalgos : string, required
maintopic : string, optional
array : int, optional
streamstojoin : string, optional
consumefrom : string, required
produceto : string, required
companyname : string, required
consumerid: string, required
producerid: string, required
inputdata: string, required
maxrows: int, optional
algokey: string, optional
partition : int, optional
offset : int, optional
hpdehost: string, required
enabletls: int, optional
delay: int, optional
hpdeport: int, required
brokerhost : string, optional
brokerport : int, optional
timeout : int, optional
usedeploy : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the prediction.
6.1 maadstml.viperhpdepredictbatch(vipertoken,host,port,consumefrom,produceto,companyname,consumerid,producerid, hpdehost,inputdata,maxrows=0,algokey='',partition=-1,offset=-1,enabletls=1,delay=1000,hpdeport=-999,brokerhost='', brokerport=-999,timeout=120,usedeploy=0,microserviceid='',topicid="-999", maintopic='', streamstojoin='', array=0,timedelay=0,asynctimeout=120,pathtoalgos='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
asynctimeout : int, optional
-This is the timeout in seconds for the Python library async function.
timedelay : int, optional
topicid : string, required
pathtoalgos : string, required
maintopic : string, optional
array : int, optional
streamstojoin : string, optional
consumefrom : string, required
produceto : string, required
companyname : string, required
consumerid: string, required
producerid: string, required
inputdata: string, required
maxrows: int, optional
algokey: string, optional
partition : int, optional
offset : int, optional
hpdehost: string, required
enabletls: int, optional
delay: int, optional
hpdeport: int, required
brokerhost : string, optional
brokerport : int, optional
timeout : int, optional
usedeploy : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the prediction.
6.2. maadstml.viperhpdepredictprocess(vipertoken,host,port,consumefrom,produceto,companyname,consumerid,producerid,hpdehost,inputdata,processtype,maxrows=0, algokey='',partition=-1,offset=-1,enabletls=1,delay=1000,hpdeport=-999,brokerhost='',brokerport=9092, timeout=120,usedeploy=0,microserviceid='',topicid=-999, maintopic='', streamstojoin='',array=0,pathtoalgos='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topicid : int, optional
pathtoalgos : string, required
maintopic : string, optional
array : int, optional
streamstojoin : string, optional
consumefrom : string, required
produceto : string, required
companyname : string, required
consumerid: string, required
producerid: string, required
inputdata: string, required
processtype: string, required
maxrows: int, optional
algokey: string, optional
partition : int, optional
offset : int, optional
hpdehost: string, required
enabletls: int, optional
delay: int, optional
hpdeport: int, required
brokerhost : string, optional
brokerport : int, optional
timeout : int, optional
usedeploy : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the prediction.
7. maadstml.viperhpdeoptimize(vipertoken,host,port,consumefrom,produceto,companyname,consumerid,producerid, hpdehost,partition=-1,offset=-1,enabletls=0,delay=100,hpdeport=-999,usedeploy=0,ismin=1,constraints='best', stretchbounds=20,constrainttype=1,epsilon=10,brokerhost='',brokerport=-999,timeout=120,microserviceid='',topicid=-999)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
consumefrom : string, required
topicid : int, optional
produceto : string, required
companyname : string, required
consumerid: string, required
producerid: string, required
hpdehost: string, required
partition : int, optional
offset : int, optional
enabletls: int, optional
delay: int, optional
hpdeport: int, required
usedeploy : int, optional
ismin : int, optional
constraints: string, optional
stretchbounds: int, optional
constrainttype: int, optional
epsilon: int, optional
brokerhost : string, optional
brokerport : int, optional
timeout : int, optional
Number of seconds that VIPER waits when trying to make a connection to HPDE.
microserviceid : string, optional
RETURNS: Returns a JSON object of the optimization details and optimal values.
7.1 maadstml.viperhpdeoptimizebatch(vipertoken,host,port,consumefrom,produceto,companyname,consumerid,producerid, hpdehost,partition=-1,offset=-1,enabletls=0,delay=100,hpdeport=-999,usedeploy=0,ismin=1,constraints='best', stretchbounds=20,constrainttype=1,epsilon=10,brokerhost='',brokerport=-999,timeout=120,microserviceid='',topicid="-999", timedelay=0,asynctimeout=120)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
asynctimeout : int, optional
-This is the timeout in seconds for the Python library async function.
timedelay : int, optional
consumefrom : string, required
topicid : string, required
produceto : string, required
companyname : string, required
consumerid: string, required
producerid: string, required
hpdehost: string, required
partition : int, optional
offset : int, optional
enabletls: int, optional
delay: int, optional
hpdeport: int, required
usedeploy : int, optional
ismin : int, optional
constraints: string, optional
stretchbounds: int, optional
constrainttype: int, optional
epsilon: int, optional
brokerhost : string, optional
brokerport : int, optional
timeout : int, optional
Number of seconds that VIPER waits when trying to make a connection to HPDE.
microserviceid : string, optional
RETURNS: Returns a JSON object of the optimization details and optimal values.
8. maadstml.viperhpdetraining(vipertoken,host,port,consumefrom,produceto,companyname,consumerid,producerid, hpdehost,viperconfigfile,enabletls=1,partition=-1,deploy=0,modelruns=50,modelsearchtuner=80,hpdeport=-999, offset=-1,islogistic=0,brokerhost='', brokerport=-999,timeout=120,microserviceid='',topicid=-999,maintopic='', independentvariables='',dependentvariable='',rollbackoffsets=0,fullpathtotrainingdata='',processlogic='', identifier='',array=0,transformtype='',sendcoefto='',coeftoprocess='',coefsubtopicnames='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
transformtype : string, optional
sendcoefto : string, optional
coeftoprocess : string, optional
coefsubtopicnames : string, optional
topicid : int, optional
array : int, optional
maintopic : string, optional
independentvariables : string, optional
dependentvariable : string, optional
rollbackoffsets: int, optional
fullpathtotrainingdata: string, optional
processlogic : string, optional
You can dynamically build a classification model by specifying how you want to classify the dependent variable by indicating your conditions in the processlogic variable (this will take effect if islogistic=1). For example:
processlogic='classification_name=my_prob:temperature=20.5,30:humidity=50,55', means the following:
This allows you to classify the dependent with any number of variables all in real-time!
consumefrom : string, required
produceto : string, required
companyname : string, required
consumerid: string, required
identifier: string, optional
You can add any name or identifier like DSN ID
Consumerid associated with the topic to consume from
producerid: string, required
hpdehost: string, required
viperconfigfile : string, required
enabletls: int, optional
partition: int, optional
deploy: int, optional
modelruns: int, optional
modelsearchtuner: int, optional
hpdeport: int, required
offset : int, optional
islogistic: int, optional
brokerhost : string, optional
brokerport : int, optional
timeout : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the optimal algorithm that best fits your data.
8.1 maadstml.viperhpdetrainingbatch(vipertoken,host,port,consumefrom,produceto,companyname,consumerid,producerid, hpdehost,viperconfigfile,enabletls=1,partition=-1,deploy=0,modelruns=50,modelsearchtuner=80,hpdeport=-999, offset=-1,islogistic=0,brokerhost='', brokerport=-999,timeout=120,microserviceid='',topicid="-999",maintopic='', independentvariables='',dependentvariable='',rollbackoffsets=0,fullpathtotrainingdata='',processlogic='', identifier='',array=0,timedelay=0,asynctimeout=120)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
asynctimeout : int, optional
-This is the timeout in seconds for the Python library async function.
timedelay : int, optional
topicid : string, required
array : int, optional
maintopic : string, optional
independentvariables : string, optional
dependentvariable : string, optional
rollbackoffsets: int, optional
fullpathtotrainingdata: string, optional
processlogic : string, optional
You can dynamically build a classification model by specifying how you want to classify the dependent variable by indicating your conditions in the processlogic variable (this will take effect if islogistic=1). For example:
processlogic='classification_name=my_prob:temperature=20.5,30:humidity=50,55', means the following:
This allows you to classify the dependent with any number of variables all in real-time!
consumefrom : string, required
produceto : string, required
companyname : string, required
consumerid: string, required
identifier: string, optional
You can add any name or identifier like DSN ID
Consumerid associated with the topic to consume from
producerid: string, required
hpdehost: string, required
viperconfigfile : string, required
enabletls: int, optional
partition: int, optional
deploy: int, optional
modelruns: int, optional
modelsearchtuner: int, optional
hpdeport: int, required
offset : int, optional
islogistic: int, optional
brokerhost : string, optional
brokerport : int, optional
timeout : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the optimal algorithm that best fits your data.
9. maadstml.viperproducetotopicstream(vipertoken,host,port,topic,producerid,offset,maxrows=0,enabletls=0,delay=100, brokerhost='',brokerport=-999,microserviceid='',topicid=-999,mainstreamtopic='',streamstojoin='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
topicid : int, optional
mainstreamtopic: string, optional
streamstojoin: string, optional
producerid : string, required
offset : int
maxrows : int, optional
enabletls: int, optional
delay: int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the optimal algorithm that best fits your data.
10. maadstml.vipercreatetrainingdata(vipertoken,host,port,consumefrom,produceto,dependentvariable, independentvariables,consumerid,producerid,companyname,partition=-1,enabletls=0,delay=100, brokerhost='',brokerport=-999,microserviceid='',topicid=-999)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
consumefrom : string, required
topicid : int, optional
produceto : string, required
dependentvariable : string, required
independentvariables : string, required
consumerid : string, required
producerid : string, required
partition : int, optional
companyname : string, required
enabletls: int, optional
delay: int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the training data set.
11. maadstml.vipercreatetopic(vipertoken,host,port,topic,companyname,contactname,contactemail,location, description,enabletls=0,brokerhost='',brokerport=-999,numpartitions=1,replication=1,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
companyname : string, required
contactname : string, required
contactemail : string, required
location : string, required
description : string, required
enabletls : int, optional
brokerhost : string, optional
brokerport : int, optional
numpartitions: int, optional
replication: int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the producer id for the topic.
12. maadstml.viperconsumefromstreamtopic(vipertoken,host,port,topic,consumerid,companyname,partition=-1, enabletls=0,delay=100,offset=0,brokerhost='',brokerport=-999,microserviceid='',topicid=-999)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
topicid : int, optional
consumerid : string, required
companyname : string, required
partition: int, optional
enabletls: int, optional
delay: int, optional
offset : int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the contents of all the topics read
13. maadstml.vipercreatejointopicstreams(vipertoken,host,port,topic,topicstojoin,companyname,contactname,contactemail, description,location,enabletls=0,brokerhost='',brokerport=-999,replication=1,numpartitions=1,microserviceid='', topicid=-999)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
topicid : int, optional
topicstojoin : string, required
companyname : string, required
contactname : string, required
contactemail : string, required
location : string, required
description : string, required
enabletls: int, optional
brokerhost : string, optional
brokerport : int, optional
numpartitions : int, optional
replication : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the producerid of the joined streams
14. maadstml.vipercreateconsumergroup(vipertoken,host,port,topic,groupname,companyname,contactname,contactemail, description,location,enabletls=1,brokerhost='',brokerport=-999,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
groupname : string, required
companyname : string, required
contactname : string, required
contactemail : string, required
location : string, required
enabletls: int, optional
description : string, required
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the groupid of the group.
15. maadstml.viperconsumergroupconsumefromtopic(vipertoken,host,port,topic,consumerid,groupid,companyname, partition=-1,enabletls=0,delay=100,offset=0,rollbackoffset=0,brokerhost='',brokerport=-999,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
consumerid : string, required
groupid : string, required
companyname : string, required
partition: int, optional
enabletls: int, optional
delay: int, optional
offset : int, optional
rollbackoffset : int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the contents of the group.
16. maadstml.vipermodifyconsumerdetails(vipertoken,host,port,topic,companyname,consumerid,contactname='', contactemail='',location='',brokerhost='',brokerport=9092,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
consumerid : string, required
companyname : string, required
contactname : string, optional
contactemail : string, optional
location : string, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: Returns success/failure
17. maadstml.vipermodifytopicdetails(vipertoken,host,port,topic,companyname,partition=0,enabletls=1, isgroup=0,contactname='',contactemail='',location='',brokerhost='',brokerport=9092,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
companyname : string, required
partition : int, optional
enabletls : int, optional
isgroup : int, optional
contactname : string, optional
contactemail : string, optional
location : string, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: Returns success/failure
18. maadstml.viperactivatetopic(vipertoken,host,port,topic,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
microserviceid : string, optional
RETURNS: Returns success/failure
19. maadstml.viperdeactivatetopic(vipertoken,host,port,topic,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
microserviceid : string, optional
RETURNS: Returns success/failure
20. maadstml.vipergroupactivate(vipertoken,host,port,groupname,groupid,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
groupname : string, required
groupid : string, required
microserviceid : string, optional
RETURNS: Returns success/failure
21. maadstml.vipergroupdeactivate(vipertoken,host,port,groupname,groupid,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
groupname : string, required
groupid : string, required
microserviceid : string, optional
RETURNS: Returns success/failure
22. maadstml.viperdeletetopics(vipertoken,host,port,topic,enabletls=1,brokerhost='',brokerport=9092,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
enabletls : int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
23. maadstml.balancebigdata(localcsvfile,numberofbins,maxrows,outputfile,bincutoff,distcutoff,startcolumn=0)
Parameters:
localcsvfile : string, required
numberofbins : int, required
maxrows : int, required
outputfile : string, required
bincutoff : float, required.
distcutoff : float, required.
startcolumn : int, optional
RETURNS: Returns a detailed JSON object and new balaced dataset written to outputfile.
24. maadstml.viperanomalytrain(vipertoken,host,port,consumefrom,produceto,producepeergroupto,produceridpeergroup,consumeridproduceto, streamstoanalyse,companyname,consumerid,producerid,flags,hpdehost,viperconfigfile, enabletls=1,partition=-1,hpdeport=-999,topicid=-999,maintopic='',rollbackoffsets=0,fullpathtotrainingdata='', brokerhost='',brokerport=9092,delay=1000,timeout=120,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
consumefrom : string, required
produceto : string, required
topicid : int, optional
maintopic : string, optional
rollbackoffsets: int, optional
fullpathtotrainingdata: string, optional
producepeergroupto : string, required
produceridpeergroup : string, required
consumeridproduceto : string, required
streamstoanalyse : string, required
flags : string, required
These are flags that will be used to select the peer group for each stream. The flags must have the following format: topic=[topic name],topictype=[numeric or string],threshnumber=[a number between 0 and 10000, i.e. 200], lag=[a number between 1 and 20, i.e. 5],zthresh=[a number between 1 and 5, i.e. 2.5],influence=[a number between 0 and 1 i.e. 0.5]
threshnumber: decimal number to determine usual behaviour - only for numeric streams, numbers are compared to the centroid number, a standardized distance is taken and all numbers below the thresholdnumeric are deemed as usual i.e. thresholdnumber=200, any value below is close to the centroid - you need to experiment with this number.
lag: number of lags for the moving mean window, works to smooth the function i.e. lag=5
zthresh: number of standard deviations from moving mean i.e. 3.5
influence: strength in identifying outliers for both stationary and non-stationary data, i.e. influence=0 ignores outliers when recalculating the new threshold, influence=1 is least robust. Influence should be between (0,1), i.e. influence=0.5
Flags must be provided for each topic. Separate multiple flags by ~
companyname : string, required
consumerid: string, required
producerid: string, required
hpdehost: string, required
viperconfigfile : string, required
enabletls: int, optional
partition: int, optional
hpdeport: int, required
brokerhost : string, optional
brokerport : int, optional
delay : int, optional
timeout : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the peer groups for all the streams.
24.1 maadstml.viperanomalytrainbatch(vipertoken,host,port,consumefrom,produceto,producepeergroupto,produceridpeergroup,consumeridproduceto, streamstoanalyse,companyname,consumerid,producerid,flags,hpdehost,viperconfigfile, enabletls=1,partition=-1,hpdeport=-999,topicid="-999",maintopic='',rollbackoffsets=0,fullpathtotrainingdata='', brokerhost='',brokerport=9092,delay=1000,timeout=120,microserviceid='',timedelay=0,asynctimeout=120)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
asynctimeout : int, optional
-This is the timeout in seconds for the Python library async function.
timedelay : int, optional
consumefrom : string, required
produceto : string, required
topicid : string, required
maintopic : string, optional
rollbackoffsets: int, optional
fullpathtotrainingdata: string, optional
producepeergroupto : string, required
produceridpeergroup : string, required
consumeridproduceto : string, required
streamstoanalyse : string, required
flags : string, required
These are flags that will be used to select the peer group for each stream. The flags must have the following format: topic=[topic name],topictype=[numeric or string],threshnumber=[a number between 0 and 10000, i.e. 200], lag=[a number between 1 and 20, i.e. 5],zthresh=[a number between 1 and 5, i.e. 2.5],influence=[a number between 0 and 1 i.e. 0.5]
threshnumber: decimal number to determine usual behaviour - only for numeric streams, numbers are compared to the centroid number, a standardized distance is taken and all numbers below the thresholdnumeric are deemed as usual i.e. thresholdnumber=200, any value below is close to the centroid - you need to experiment with this number.
lag: number of lags for the moving mean window, works to smooth the function i.e. lag=5
zthresh: number of standard deviations from moving mean i.e. 3.5
influence: strength in identifying outliers for both stationary and non-stationary data, i.e. influence=0 ignores outliers when recalculating the new threshold, influence=1 is least robust. Influence should be between (0,1), i.e. influence=0.5
Flags must be provided for each topic. Separate multiple flags by ~
companyname : string, required
consumerid: string, required
producerid: string, required
hpdehost: string, required
viperconfigfile : string, required
enabletls: int, optional
partition: int, optional
hpdeport: int, required
brokerhost : string, optional
brokerport : int, optional
delay : int, optional
timeout : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the peer groups for all the streams.
25. maadstml.viperanomalypredict(vipertoken,host,port,consumefrom,produceto,consumeinputstream,produceinputstreamtest,produceridinputstreamtest, streamstoanalyse,consumeridinputstream,companyname,consumerid,producerid,flags,hpdehost,viperconfigfile, enabletls=1,partition=-1,hpdeport=-999,topicid=-999,maintopic='',rollbackoffsets=0,fullpathtopeergroupdata='', brokerhost='',brokerport=9092,delay=1000,timeout=120,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
consumefrom : string, required
produceto : string, required
consumeinputstream : string, required
produceinputstreamtest : string, required
produceridinputstreamtest : string, required
streamstoanalyse : string, required
flags : string, required
These are flags that will be used to select the peer group for each stream. The flags must have the following format: *riskscore=[a number between 0 and 1]~complete=[and, or, pvalue i.e. p50 means streams over 50% that have an anomaly]~type=[and,or this will determine what logic to apply to v and sc],topic=[topic name],topictype=[numeric or string],v=[v>some value, v<some value, or valueany], sc=[sc>some number, sc<some number - this is the score for the anomaly test]
if using strings, the specify flags: type=[and,or],topic=[topic name],topictype=string,stringcontains=[0 or 1 - 1 will do a substring test, 0 will equate the strings],v2=[any text you want to test - use | for OR or ^ for AND],sc=[score value, sc<some value, sc>some value]
riskscore: this the riskscore threshold. A decimal number between 0 and 1, use this as a threshold to flag anomalies.
complete : If using multiple streams, this will test each stream to see if the computed riskscore and perform an AND or OR on each risk value and take an average of the risk scores if using AND. Otherwise if at least one stream exceeds the riskscore it will return.
type: AND or OR - if using v or sc, this is used to apply the appropriate logic between v and sc. For example, if type=or, then VIPER will see if a test value is less than or greater than V, OR, standarzided value is less than or greater than sc.
sc: is a standarized variavice between the peer group value and test value.
v1: is a user chosen value which can be used to test for a particular value. For example, if you want to flag values less then 0, then choose v<0 and VIPER will flag them as anomolous.
v2: if analysing string streams, v2 can be strings you want to check for. For example, if I want to check for two
strings: Failed and Attempt Failed, then set v2=Failed^Attempt Failed, where ^ tells VIPER to perform an AND operation.
If I want either to exist, 2=Failed|Attempt Failed, where | tells VIPER to perform an OR operation.
stringcontains : if using string streams, and you want to see if a particular text value exists and flag it - then if stringcontains=1, VIPER will test for substrings, otherwise it will equate the strings.
Flags must be provided for each topic. Separate multiple flags by ~
consumeridinputstream : string, required
companyname : string, required
consumerid: string, required
producerid: string, required
hpdehost: string, required
viperconfigfile : string, required
enabletls: int, optional
partition: int, optional
hpdeport: int, required
topicid : int, optional
maintopic : string, optional
rollbackoffsets: int, optional
fullpathtopeergroupdata: string, optional
brokerhost : string, optional
brokerport : int, optional
delay : int, optional
timeout : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the peer groups for all the streams.
25.1 maadstml.viperanomalypredictbatch(vipertoken,host,port,consumefrom,produceto,consumeinputstream,produceinputstreamtest,produceridinputstreamtest, streamstoanalyse,consumeridinputstream,companyname,consumerid,producerid,flags,hpdehost,viperconfigfile, enabletls=1,partition=-1,hpdeport=-999,topicid="-999",maintopic='',rollbackoffsets=0,fullpathtopeergroupdata='', brokerhost='',brokerport=9092,delay=1000,timeout=120,microserviceid='',timedelay=0,asynctimeout=120)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
asynctimeout : int, optional
-This is the timeout in seconds for the Python library async function.
timedelay : int, optional
consumefrom : string, required
produceto : string, required
consumeinputstream : string, required
produceinputstreamtest : string, required
produceridinputstreamtest : string, required
streamstoanalyse : string, required
flags : string, required
These are flags that will be used to select the peer group for each stream. The flags must have the following format: *riskscore=[a number between 0 and 1]~complete=[and, or, pvalue i.e. p50 means streams over 50% that have an anomaly]~type=[and,or this will determine what logic to apply to v and sc],topic=[topic name],topictype=[numeric or string],v=[v>some value, v<some value, or valueany], sc=[sc>some number, sc<some number - this is the score for the anomaly test]
if using strings, the specify flags: type=[and,or],topic=[topic name],topictype=string,stringcontains=[0 or 1 - 1 will do a substring test, 0 will equate the strings],v2=[any text you want to test - use | for OR or ^ for AND],sc=[score value, sc<some value, sc>some value]
riskscore: this the riskscore threshold. A decimal number between 0 and 1, use this as a threshold to flag anomalies.
complete : If using multiple streams, this will test each stream to see if the computed riskscore and perform an AND or OR on each risk value and take an average of the risk scores if using AND. Otherwise if at least one stream exceeds the riskscore it will return.
type: AND or OR - if using v or sc, this is used to apply the appropriate logic between v and sc. For example, if type=or, then VIPER will see if a test value is less than or greater than V, OR, standarzided value is less than or greater than sc.
sc: is a standarized variavice between the peer group value and test value.
v1: is a user chosen value which can be used to test for a particular value. For example, if you want to flag values less then 0, then choose v<0 and VIPER will flag them as anomolous.
v2: if analysing string streams, v2 can be strings you want to check for. For example, if I want to check for two
strings: Failed and Attempt Failed, then set v2=Failed^Attempt Failed, where ^ tells VIPER to perform an AND operation.
If I want either to exist, 2=Failed|Attempt Failed, where | tells VIPER to perform an OR operation.
stringcontains : if using string streams, and you want to see if a particular text value exists and flag it - then if stringcontains=1, VIPER will test for substrings, otherwise it will equate the strings.
Flags must be provided for each topic. Separate multiple flags by ~
consumeridinputstream : string, required
companyname : string, required
consumerid: string, required
producerid: string, required
hpdehost: string, required
viperconfigfile : string, required
enabletls: int, optional
partition: int, optional
hpdeport: int, required
topicid : string, required
maintopic : string, optional
rollbackoffsets: int, optional
fullpathtopeergroupdata: string, optional
brokerhost : string, optional
brokerport : int, optional
delay : int, optional
timeout : int, optional
microserviceid : string, optional
RETURNS: Returns a JSON object of the peer groups for all the streams.
26. maadstml.viperpreprocessproducetotopicstream(VIPERTOKEN,host,port,topic,producerid,offset,maxrows=0,enabletls=0,delay=100, brokerhost='',brokerport=-999,microserviceid='',topicid=-999,streamstojoin='',preprocesslogic='', preprocessconditions='',identifier='',preprocesstopic='',array=0,saveasarray=0,rawdataoutput=0)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
array : int, optional
rawdataoutput : int, optional
preprocessconditions : string, optional
You can set conditions to aggregate functions: MIN, MAX, AVG, COUNT, COUNTSTR, DIFF, DIFFMARGIN, SUM, MEDIAN, VARIANCE, OUTLIERS, OUTLIERSX-Y, VARIED, ANOMPROB,ANOMPROBX-Y, CONSISTENCY, ENTROPY, AUTOCORR, TREND, IQR (InterQuartileRange), Midhinge, GM (Geometric mean), HM (Harmonic mean), Trimean, CV (coefficient of Variation), Mad (Mean absolute deviation),Skewness, Kurtosis, Spikedetect, Unique, Uniquestr, Timediff: time should be in this layout:2006-01-02T15:04:05, Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the average time in seconds between consecutive dates. Spikedetect uses a Zscore method to detect spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5. Geodiff (returns distance in Kilometers between two lat/long points) Unique Checks numeric data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from current local time. You can specify the UTC offset to adjust the current time to match the timezone of the data stream. You can specify timetype as millisecond, second, minute, hour, day. For example, if Dataage_1_minute, then this processtype will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference between the data stream timestamp and current local time and return the difference in minutes. This is a very powerful processtype for data quality and data assurance programs for any number of data streams.
Uniquestr Checks string data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Uniquecount Checks numeric data for duplication. Returns count of unique numbers.
Uniquestrcount Checks string data for duplication. Returns count of unique strings.
CONSISTENCY checks if the data all have consistent data types. Returns 1 for consistent data types, 0 otherwise.
Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high
RAW for no processing.
ANOMPROB=Anomaly Probability, it will run several algorithms on the data stream window to determine a probaility of anomalous behaviour. This can be cross-refenced with OUTLIERS. It can be very powerful way to detection issues with devices. VARIED will determine if the values in the window are all the same, or varied: it will return 1 for varied, 0 if values are all the same. This is useful if you want to know if something changed in the stream.
ANOMPROBX-Y (similar to OUTLIERSX-Y), where X and Y are numbers or "n". If "n" means examine all anomalies for patterns. They allow you to check if the anomalies in the streams are truly anomalies and not some pattern. For example, if a IoT device shuts off and turns on again routinely, this may be picked up as an anomaly when in fact it is normal behaviour. So, to ignore these cases, if ANOMPROB2-5, this tells Viper, check anomalies with patterns of 2-5 peaks. If the stream has two classes and these two classes are like 0 and 1000, and show a pattern, then they should not be considered an anomaly. Meaning, class=0, is the device shutting down, class=1000 is the device turning back on. If ANOMPROB3-10, Viper will check for patterns of classes 3 to 10 to see if they recur routinely. This is very helpful to reduce false positives and false negatives.
For example, preprocessconditions='humidity=55,60:temperature=34,n', and preprocesslogic='max,count', means Get the MAX value of values in humidity if humidity is between [55,60], and Count values in temperature if temperature >=34.
preprocesstopic : string, optional
identifier : string, optional
producerid : string, required
offset : int, optional
saveasarray : int, optional
maxrows : int, optional
enabletls: int, optional
delay: int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
topicid : int, optional
streamstojoin : string, optional
preprocesslogic : string, optional
Here you need to specify how you want to pre-process the streams. You can perform the following operations: MAX, MIN, AVG, COUNT, COUNTSTR, SUM, DIFF, DIFFMARGIN, VARIANCE, MEDIAN, OUTLIERS, OUTLIERSX-Y, VARIED, ANOMPROB, ANOMPROBX-Y, ENTROPY, AUTOCORR, TREND, CONSISTENCY, Unique, Uniquestr, Geodiff (returns distance in Kilometers between two lat/long points), IQR (InterQuartileRange), Midhinge, GM (Geometric mean), HM (Harmonic mean), Trimean, CV (coefficient of Variation), Mad (Mean absolute deviation), Skewness, Kurtosis, Spikedetect, Timediff: time should be in this layout:2006-01-02T15:04:05, Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the average time in seconds between consecutive dates. Uniquecount Checks numeric data for duplication. Returns count of unique numbers.
Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from current local time. You can specify the UTC offset to adjust the current time to match the timezone of the data stream. You can specify timetype as millisecond, second, minute, hour, day. For example, if Dataage_1_minute, then this processtype will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference between the data stream timestamp and current local time and return the difference in minutes. This is a very powerful processtype for data quality and data assurance programs for any number of data streams.
Uniquestrcount Checks string data for duplication. Returns count of unique strings.
Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high
RAW for no processing.
Spikedetect uses a Zscore method to detect spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5.
The order of the operation must match the order of the stream. If you specified topicid, you can perform TML on the new preprocessed stream append appending: _preprocessed_processlogic For example, if streamstojoin="stream1,stream2,streams3", and preprocesslogic="min,max,diff", the new streams will be: stream1_preprocessed_Min, stream2_preprocessed_Max, stream3_preprocessed_Diff.
RETURNS: Returns preprocessed JSON.
27. maadstml.areyoubusy(host,port)
Parameters:
host : string, required
port : int, required
RETURNS: Returns a list of available VIPER and HPDE with their HOST and PORT.
28. maadstml.viperstreamquery(VIPERTOKEN,host,port,topic,producerid,offset=-1,maxrows=0,enabletls=1,delay=100,brokerhost='', brokerport=-999,microserviceid='',topicid=-999,streamstojoin='',preprocessconditions='', identifier='',preprocesstopic='',description='',array=0)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, required
producerid : string, required
offset : int, optional
maxrows : int, optional
enabletls: int, optional
delay: int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
topicid : int, optional
streamstojoin : string, required
preprocessconditions : string, required
identifier: string, optional
preprocesstopic : string, optional
description : string, optional
array : int, optional
RETURNS: 1 if the condition is TRUE (condition met), 0 if false (condition not met)
28.1 maadstml.viperstreamquerybatch(VIPERTOKEN,host,port,topic,producerid,offset=-1,maxrows=0,enabletls=1,delay=100,brokerhost='', brokerport=-999,microserviceid='',topicid="-999",streamstojoin='',preprocessconditions='', identifier='',preprocesstopic='',description='',array=0,timedelay=0,asynctimeout=120)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
asynctimeout : int, optional
-This is the timeout in seconds for the Python library async function.
timedelay : int, optional
topic : string, required
producerid : string, required
offset : int, optional
maxrows : int, optional
enabletls: int, optional
delay: int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
topicid : string, required
streamstojoin : string, required
preprocessconditions : string, required
identifier: string, optional
preprocesstopic : string, optional
description : string, optional
array : int, optional
RETURNS: 1 if the condition is TRUE (condition met), 0 if false (condition not met)
29. maadstml.viperpreprocessbatch(VIPERTOKEN,host,port,topic,producerid,offset,maxrows=0,enabletls=0,delay=100, brokerhost='',brokerport=-999,microserviceid='',topicid="-999",streamstojoin='',preprocesslogic='', preprocessconditions='',identifier='',preprocesstopic='',array=0,saveasarray=0,timedelay=0,asynctimeout=120,rawdataoutput=0)
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
asynctimeout : int, optional
-This is the timeout in seconds for the Python library async function.
rawdataoutput : int, optional
-Set rawdataoutput=1 to output the raw preprocessing data to the Json.
timedelay : int, optional
topic : string, required
array : int, optional
preprocessconditions : string, optional
You can set conditions to aggregate functions: MIN, MAX, AVG, COUNT, COUNTSTR, DIFF, SUM, MEDIAN, VARIANCE, OUTLIERS, OUTLIERSX-Y, VARIED, ANOMPROB,ANOMPROBX-Y, ENTROPY, AUTOCORR, TREND, IQR (InterQuartileRange), Midhinge, GM (Geometric mean), HM (Harmonic mean), Trimean, CV (coefficient of Variation), Mad (Mean absolute deviation),Skewness, Kurtosis, Spikedetect, Unique, Uniquestr, Timediff: time should be in this layout:2006-01-02T15:04:05, Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the average time in seconds between consecutive dates. Spikedetect uses a Zscore method to detect spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5. Geodiff (returns distance in Kilometers between two lat/long points).
Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from current local time. You can specify the UTC offset to adjust the current time to match the timezone of the data stream. You can specify timetype as millisecond, second, minute, hour, day. For example, if Dataage_1_minute, then this processtype will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference between the data stream timestamp and current local time and return the difference in minutes. This is a very powerful processtype for data quality and data assurance programs for any number of data streams.
Unique Checks numeric data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Uniquestr Checks string data for duplication. Returns 1 if no data duplication (unique), 0 otherwise. Uniquecount Checks numeric data for duplication. Returns count of unique numbers. Uniquestrcount Checks string data for duplication. Returns count of unique strings.
Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high
ANOMPROB=Anomaly Probability, it will run several algorithms on the data stream window to determine a probaility of anomalous behaviour. This can be cross-refenced with OUTLIERS. It can be very powerful way to detection issues with devices. VARIED will determine if the values in the window are all the same, or varied: it will return 1 for varied, 0 if values are all the same. This is useful if you want to know if something changed in the stream.
ANOMPROBX-Y (similar to OUTLIERSX-Y), where X and Y are numbers or "n". If "n" means examine all anomalies for patterns. They allow you to check if the anomalies in the streams are truly anomalies and not some pattern. For example, if a IoT device shuts off and turns on again routinely, this may be picked up as an anomaly when in fact it is normal behaviour. So, to ignore these cases, if ANOMPROB2-5, this tells Viper, check anomalies with patterns of 2-5 peaks. If the stream has two classes and these two classes are like 0 and 1000, and show a pattern, then they should not be considered an anomaly. Meaning, class=0, is the device shutting down, class=1000 is the device turning back on. If ANOMPROB3-10, Viper will check for patterns of classes 3 to 10 to see if they recur routinely. This is very helpful to reduce false positives and false negatives.
For example, preprocessconditions='humidity=55,60:temperature=34,n', and preprocesslogic='max,count', means Get the MAX value of values in humidity if humidity is between [55,60], and Count values in temperature if temperature >=34.
preprocesstopic : string, optional
identifier : string, optional
producerid : string, required
offset : int, optional
saveasarray : int, optional
maxrows : int, optional
enabletls: int, optional
delay: int, optional
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
topicid : string, required
streamstojoin : string, optional
preprocesslogic : string, optional
Here you need to specify how you want to pre-process the streams. You can perform the following operations: MAX, MIN, AVG, COUNT, COUNTSTR, SUM, DIFF, VARIANCE, MEDIAN, OUTLIERS, OUTLIERSX-Y, VARIED, ANOMPROB, ANOMPROBX-Y, ENTROPY, AUTOCORR, TREND, IQR (InterQuartileRange), Midhinge, CONSISTENCY, GM (Geometric mean), HM (Harmonic mean), Trimean, CV (coefficient of Variation), Mad (Mean absolute deviation), Skewness, Kurtosis, Spikedetect, Unique, Uniquestr, Timediff: time should be in this layout:2006-01-02T15:04:05, Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the average time in seconds between consecutive dates. Geodiff (returns distance in Kilometers between two lat/long points). Spikedetect uses a Zscore method to detect spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5. Uniquecount Checks numeric data for duplication. Returns count of unique numbers. Uniquestrcount Checks string data for duplication. Returns count of unique strings.
Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from current local time. You can specify the UTC offset to adjust the current time to match the timezone of the data stream. You can specify timetype as millisecond, second, minute, hour, day. For example, if Dataage_1_minute, then this processtype will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference between the data stream timestamp and current local time and return the difference in minutes. This is a very powerful processtype for data quality and data assurance programs for any number of data streams.
Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high
The order of the operation must match the order of the stream. If you specified topicid, you can perform TML on the new preprocessed stream append appending: _preprocessed_processlogic For example, if streamstojoin="stream1,stream2,streams3", and preprocesslogic="min,max,diff", the new streams will be: stream1_preprocessed_Min, stream2_preprocessed_Max, stream3_preprocessed_Diff.
RETURNS: None.
30. maadstml.viperlisttopics(vipertoken,host,port=-999,brokerhost='', brokerport=-999,microserviceid='')
Parameters:
VIPERTOKEN : string, required
host : string, required
port : int, required
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: A JSON formatted object of all the topics in the Kafka broker.
31. maadstml.viperpreprocesscustomjson(VIPERTOKEN,host,port,topic,producerid,offset,jsoncriteria='',rawdataoutput=0,maxrows=0, enabletls=0,delay=100,brokerhost='',brokerport=-999,microserviceid='',topicid=-999,streamstojoin='',preprocesslogic='', preprocessconditions='',identifier='',preprocesstopic='',array=0,saveasarray=0,timedelay=0,asynctimeout=120, usemysql=0,tmlfilepath='',pathtotmlattrs='')
Parameters:
VIPERTOKEN : string, required
host : string, required
topic : string, required
producerid : string, required
offset : int, required
jsoncriteria : string, required
This is the JSON path to the data you want to consume . It must be the following format:
*UID* is path to the main id. For example, Patient ID
*filter* is the path to something that filter the jsons
*subtopic* is the path to the subtopics in the json (several paths can be specified)
*values* is the path to the Values of the subtopics - Subtopic and Value must have 1-1 match
*identifiers* is the path to any special identifiers for the subtopics
*datetime* is the path to the datetime of the message
*msgid* is the path to any msg id
For example:
jsoncriteria='uid=subject.reference,filter:resourceType=Observation~\
subtopics=code.coding.0.code,component.0.code.coding.0.code,component.1.code.coding.0.code~\
values=valueQuantity.value,component.0.valueQuantity.value,component.1.valueQuantity.value~\
identifiers=code.coding.0.display,component.0.code.coding.0.display,component.1.code.coding.0.display~\
datetime=effectiveDateTime~\
msgid=id'
rawdataoutput : int, optional
maxrows : int, optional
enabletls : int, optional
delay : int, optional
topicid : int, optional
streamstojoin : string, optional
preprocesslogic : string, optional
Specify your preprocess algorithms. For example, You can set conditions to aggregate functions: MIN, MAX, AVG, COUNT, COUNTSTR, DIFF, DIFFMARGIN, SUM, MEDIAN, VARIANCE, OUTLIERS, OUTLIERSX-Y, VARIED, ANOMPROB,ANOMPROBX-Y, CONSISTENCY, ENTROPY, AUTOCORR, TREND, IQR (InterQuartileRange), Midhinge, GM (Geometric mean), HM (Harmonic mean), Trimean, CV (coefficient of Variation), Mad (Mean absolute deviation),Skewness, Kurtosis, Spikedetect, Unique, Uniquestr, Timediff: time should be in this layout:2006-01-02T15:04:05, Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the average time in seconds between consecutive dates. Spikedetect uses a Zscore method to detect spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5. Geodiff (returns distance in Kilometers between two lat/long points) Unique Checks numeric data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from current local time. You can specify the UTC offset to adjust the current time to match the timezone of the data stream. You can specify timetype as millisecond, second, minute, hour, day. For example, if Dataage_1_minute, then this processtype will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference between the data stream timestamp and current local time and return the difference in minutes. This is a very powerful processtype for data quality and data assurance programs for any number of data streams.
Uniquestr Checks string data for duplication. Returns 1 if no data duplication (unique), 0 otherwise.
Uniquecount Checks numeric data for duplication. Returns count of unique numbers.
Uniquestrcount Checks string data for duplication. Returns count of unique strings.
CONSISTENCY checks if the data all have consistent data types. Returns 1 for consistent data types, 0 otherwise.
Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high
RAW for no processing.
preprocessconditions : string, optional
identifier : string, optional
preprocesstopic : string, optional
array : int, optional
saveasarray : int, optional
timedelay : int, optional
asynctimeout : int, optional
usemysql : int, optional
tmlfilepath : string, optional
pathtotmlattrs : string, optional
Specifiy any attributes for the TMLID. Here you can specify OEM, Latitude, Longitude, and Location JSON paths:
pathtotmlattrs='oem=id,lat=subject.reference,long=component.0.code.coding.0.display,location=component.1.valueQuantity.value'
port : int, required
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: null
32. maadstml.viperstreamcorr(vipertoken,host,port,topic,producerid,offset=-1,maxrows=0,enabletls=1,delay=100,brokerhost='', brokerport=-999,microserviceid='',topicid=-999,streamstojoin='', identifier='',preprocesstopic='',description='',array=0, wherecondition='', wheresearchkey='PreprocessIdentifier',rawdataoutput=1,threshhold=0,pvalue=0, identifierextractpos="",topcorrnum=5,jsoncriteria='',tmlfilepath='',usemysql=0, pathtotmlattrs='',mincorrvectorlen=5,writecorrstotopic='',outputtopicnames=0,nlp=0, correlationtype='',docrosscorr=0)
Parameters: Perform Stream correlations
VIPERTOKEN : string, required
host : string, required
topic : string, required
producerid : string, required
wherecondition : string, optional
correlationtype : string, optional
docrosscorr : int, optional
wheresearchkey : string, optional
description : string, optional
identifierextractpos : string, optional
offset : int, required
mincorrvectorlen : int, optional
topcorrnum : int, optional
threshhold : int, optional
pvalue : int, optional
writecorrstotopic : string, optional
outputtopicnames : int, optional
nlp : int, optional
jsoncriteria : string, required
This is the JSON path to the data you want to consume . It must be the following format:
*UID* is path to the main id. For example, Patient ID
*filter* is the path to something that filter the jsons
*subtopic* is the path to the subtopics in the json (several paths can be specified)
*values* is the path to the Values of the subtopics - Subtopic and Value must have 1-1 match
*identifiers* is the path to any special identifiers for the subtopics
*datetime* is the path to the datetime of the message
*msgid* is the path to any msg id
For example:
jsoncriteria='uid=subject.reference,filter:resourceType=Observation~\
subtopics=code.coding.0.code,component.0.code.coding.0.code,component.1.code.coding.0.code~\
values=valueQuantity.value,component.0.valueQuantity.value,component.1.valueQuantity.value~\
identifiers=code.coding.0.display,component.0.code.coding.0.display,component.1.code.coding.0.display~\
datetime=effectiveDateTime~\
msgid=id'
rawdataoutput : int, optional
maxrows : int, optional
enabletls : int, optional
delay : int, optional
topicid : int, optional
streamstojoin : string, optional
preprocesslogic : string, optional
preprocessconditions : string, optional
identifier : string, optional
preprocesstopic : string, optional
array : int, optional
saveasarray : int, optional
timedelay : int, optional
asynctimeout : int, optional
usemysql : int, optional
tmlfilepath : string, optional
pathtotmlattrs : string, optional
Specifiy any attributes for the TMLID. Here you can specify OEM, Latitude, Longitude, and Location JSON paths:
pathtotmlattrs='oem=id,lat=subject.reference,long=component.0.code.coding.0.display,location=component.1.valueQuantity.value'
port : int, required
brokerhost : string, optional
brokerport : int, optional
microserviceid : string, optional
RETURNS: null
33. maadstml.viperstreamcluster(vipertoken,host,port,topic,producerid,offset=-1,maxrows=0,enabletls=1,delay=100,brokerhost='', brokerport=-999,microserviceid='',topicid=-999,iterations=1000, numclusters=8, distancealgo=1,description='',rawdataoutput=0,valuekey='',filterkey='',groupkey='', identifier='',datetimekey='',valueidentifier='',msgid='',valuecondition='', identifierextractpos='',preprocesstopic='', alertonclustersize=0,alertonsubjectpercentage=50,sendalertemailsto='',emailfrequencyinseconds=0, companyname='',analysisdescription='',identifierextractposlatitude=-1, identifierextractposlongitude=-1,identifierextractposlocation=-1, identifierextractjoinedidentifiers=-1,pdfformat='',minimumsubjects=2)
Parameters: Perform Stream correlations
VIPERTOKEN : string, required
host : string, required
topic : string, required
port : int, required
brokerhost : string, optional
brokerport : int, optional
alertonsubjectpercentage : int, optional
identifierextractjoinedidentifiers : int, optional
pdfformat : string, optional
Speficy format text of the PDF to generate and emailed to users. You can set title, signature, showpdfemaillist, and charttitle.
pdfformat="title=This is a Transactional Machine Learning Auto-Generated PDF for Cluster Analysis For OTICS|signature=
Created by: OTICS, Toronto|showpdfemaillist=1|charttitle=Chart Shows Clusters of Patients with Similar Symptoms"
minimumsubjects : int, optional
microserviceid : string, optional
maxrows : int, optional
enabletls : int, optional
delay : int, optional
producerid : string, required
topicid : int, optional
iterations : int, optional
numclusters : int, optional
distancealgo : int, optional
valuekey : string, required
filterkey : string, optional
groupkey : string, optional
valueidentifier : string, optional
msgid : string, optional
valuecondition : string, optional
identifierextractpos : string, optional
preprocesstopic : string, required
alertonclustersize : int, optional
sendalertemailsto: string, optional
emailfrequencyinseconds : int, optional
companyname : string, optional
analysisdescription : string, optional
identifierextractposlatitude : int, optional
identifierextractposlongitude : int, optional
identifierextractposlocation : int, optional
RETURNS: null
34. maadstml.vipersearchanomaly(vipertoken,host,port,topic,producerid,offset,jsoncriteria='',rawdataoutput=0,maxrows=0,enabletls=0,delay=100, brokerhost='',brokerport=-999,microserviceid='',topicid=-999,identifier='',preprocesstopic='', timedelay=0,asynctimeout=120,searchterms='',entitysearch='',tagsearch='',checkanomaly=1,testtopic='', includeexclude=1,anomalythreshold=0,sendanomalyalertemail='',emailfrequency=3600)
Parameters: Perform Stream correlations
VIPERTOKEN : string, required
host : string, required
topic : string, required
port : int, required
brokerhost : string, optional
brokerport : int, optional
jsoncriteria : string, optional
anomalythreshold : int, optional
includeexclude : int, optional
sendanomalyalertemail : string, optional
microserviceid : string, optional
maxrows : int, optional
enabletls : int, optional
delay : int, optional
producerid : string, required
emailfrequency : int, optional
testtopic : string, optional
preprocesstopic : string, required
sendalertemailsto: string, optional
tagsearch : string, optional
entitysearch : string, optional
searchterms : string, optional
emailfrequencyinseconds : int, optional
companyname : string, optional
topicid : int, optional
identifier : string, optional
checkanomaly : int, optional
rawdataoutput : int, optional
RETURNS: null
35. maadstml.vipermirrorbrokers(VIPERTOKEN,host,port,brokercloudusernamepassfrom,brokercloudusernamepassto,
enabletlsfrom,enabletlsto,
replicationfactorfrom,replicationfactorto,compressionfrom,compressionto,
saslfrom,saslto,partitions,brokerlistfrom,brokerlistto,
topiclist,asynctimeout=300,microserviceid="",servicenamefrom="broker",
servicenameto="broker",partitionchangeperc=0,replicationchange=0,filter="",rollbackoffset=0)
Parameters: Perform Data Stream migration across brokers - fast and simple.
VIPERTOKEN : string, required
host : string, required
port : int, required
brokercloudusernamepassfrom : string, required
brokercloudusernamepassto : string, required
enabletlsfrom : string, required
enabletlsto : string, required
replicationfactorfrom : string, optional
replicationfactorto : string, optional
compressionfrom : string, required
compressionto : string, required
saslfrom : string, required
saslto : string, required
partitions : string, optional
brokerlistfrom : string, required
brokerlistto : string, required
topiclist : string, optional
partitionchangeperc : number, optional
replicationchange : ignored for now
filter : string, optional
asynctimeout : number, optional
microserviceid : string, optional
servicenamefrom : string, optional
servicenameto : string, optional
rollbackoffset: ignored
36. maadstml.vipernlp(filename,maxsummarywords,maxkeywords)
Parameters: Perform NLP summarization of PDFs
filename : string, required
maxsummarywords : int, required
maxkeywords : int, required
RETURNS: JSON string of summary.
37. maadstml.viperchatgpt(openaikey,texttoanalyse,query, temperature,modelname)
Parameters: Start a conversation with ChatGPT
openaikey : string, required
texttoanalyse : string, required
query : string, required
temperature : float, required
modelname : string, required
RETURNS: ChatGPT response.
38. maadstml.viperexractpdffields(pdffilename)
Parameters: Extract data from PDF
pdffilename : string, required
RETURNS: JSON of PDF and writes JSON and XML files of PDF to disk.
39. maadstml.viperexractpdffieldbylabel(pdffilename,labelname,arcotype)
Parameters: Extract data from PDF by PDF labels
pdffilename : string, required
labelname : string, required
pdffilename,labelname,arcotype : string, required
RETURNS: Value of the labelname - if any.
40. maadstml.pgptingestdocs(docname,doctype, pgptip,pgptport,pgptendpoint)
Parameters:
docname : string, required
doctype : string, required
pgptip : string, required
pgptport : string, required
pgptendpoint : string, required
RETURNS: JSON containing Document details, or ERROR.
41. maadstml.pgptgetingestedembeddings(docname,ip,port,endpoint)
Parameters:
docname : string, required
ip : string, required
port : string, required
endpoint : string, required
RETURNS: Three variables: docids,docstr,docidsstr; these are the embeddings related to docname. Or, ERROR.
42. maadstml.pgptchat(prompt,context,docfilter,port,includesources,ip,endpoint)
Parameters:
prompt : string, required
context : bool, required
docfilter : string array, required
port : string, required
includesources : bool, required
ip : string, required
endpoint : string, required
RETURNS: The response from privateGPT, or ERROR.
43. maadstml.pgptdeleteembeddings(docids, ip,port,endpoint)
Parameters:
docids : string array, required
port : string, required
ip : string, required
endpoint : string, required
RETURNS: Null if successful, or ERROR.
44. maadstml.pgpthealth(ip,port,endpoint)
Parameters:
port : string, required
ip : string, required
endpoint : string, required
RETURNS: This will return a JSON of OK if the privateGPT server is running, or ERROR.
45. maadstml.videochatloadresponse(url,port,filename,prompt,responsefolder='videogpt_response',temperature=0.2,max_output_tokens=512)
Parameters:
url : string, required
port : string, required
filename : string, required
prompt : string, required
responsefolder : string, optional
temperature : float, optional
max_output_tokens : int, optional
RETURNS: The file name the response was written to by video chatgpt.
46. maadstml.viperpreprocessrtms(vipertoken,host,port,topic,producerid,offset,maxrows=0,enabletls=0,delay=100,brokerhost='',brokerport=-999,microserviceid='', topicid=-999,rtmsstream='',searchterms='',rememberpastwindows='',identifier='', preprocesstopic='',patternwindowthreshold='',array=0,saveasarray=0,rawdataoutput=0, rtmsscorethreshold='',rtmsscorethresholdtopic='',attackscorethreshold='', attackscorethresholdtopic='',patternscorethreshold='',patternscorethresholdtopic='',rtmsmaxwindows='10000'):
Parameters:
** : string, required
VIPERTOKEN : string, required
host : string, required
port : int, required
topic : string, optional
producerid : string, required
offset : int, required
maxrows : int, required
enabletls : int, required
brokerhost : string, optional
brokerport : int, optional
delay : int, optional
microserviceid : string, optional
topicid : int, required
rtmsstream : string, required
searchterms : string, required
Search terms to use to search the data in text data. Separate by semi-colon for
for different searches for different rtmsstream topics.
rememberpastwindows : int, required
identifier : string, optional
preprocesstopic : string, required
patternwindowthreshold : int, required
Threshold number of windows for the occurence
of patterns of the search terms.
rtmsscorethreshold: string, optional
rtmsscorethresholdtopic: string, optional
Name of a kafka topic that will contain messages greater
than rtmsthreshold
attackscorethreshold: string, optional
attackscorethresholdtopic: string, optional
Name of a kafka topic that will contain messages greater
than attackthreshold
patternscorethreshold: string, optional
patternscorethresholdtopic: string, optional
Name of a kafka topic that will contain messages greater
than patternthreshold
array : int, optional
saveasarray : int, optional
rawdataoutput : int, optional
rtmsmaxwindows : int, optional
RETURNS: Null
FAQs
Multi-Agent Accelerator for Data Science (MAADS): Transactional Machine Learning
We found that maadstml demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
OpenGrep has restored fingerprint and metavariable support in JSON and SARIF outputs, making static analysis more effective for CI/CD security automation.
Security News
Security experts warn that recent classification changes obscure the true scope of the NVD backlog as CVE volume hits all-time highs.
Security Fundamentals
Attackers use obfuscation to hide malware in open source packages. Learn how to spot these techniques across npm, PyPI, Maven, and more.