Siwi (/ΛsΙͺwi/) is a PoC of Dialog System With Graph Database Backed Knowledge Graph.
For now, it's a demo for task-driven(not general purpose) dialog bots with KG(Knowledge Graph) leveraging Nebula Graph with the minimal/sample dataset from Nebula Graph Manual/ NGδΈζζε.
Tips: Now you can play with the graph online without installing yourself!
Supported queries:
relation:
serving:
friendship:
You can try with it from scratch here: https://katacoda.com/wey/scenarios/siwi-kgqa
This is one of the most naive pipeline for a specific domain/ single purpose chat bot built on a Knowledge Graph.
The Backend(Siwi API) is a Flask based API server:
Flask API server takes questions in HTTP POST, and calls the bot API.
In bot API part there are classfier(Symentic Parsing, Intent Matching, Slot Filling), and question actors(Call corresponding actions to query Knowledge Graph with intents and slots).
Knowledge Graph is built on an Open-Source Graph Database: Nebula Graph
The Frontend is a VueJS Single Page Applicaiton(SPA):
ββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββ
β β β
β β Speech β
β ββββββββββββΌβββββββββββ β
β β Frontend β Siwi, /ΛsΙͺwi/ β
β β Web_Speech_API β A PoC of β
β β β Dialog System β
β β Vue.JS β With Graph Database β
β β β Backed Knowledge Graph β
β ββββββββββββ¬βββββββββββ β
β β Sentence β
β β β
β ββββββββββββββΌβββββββββββββββββββββββββββββββ β
β β β β β
β β β Backend β β
β β ββββββββββββΌβββββββββββ β β
β β β Web API, Flask β ./app/ β β
β β ββββββββββββ¬βββββββββββ β β
β β β Sentence ./bot/ β β
β β ββββββββββββΌβββββββββββ β β
β β β β β β
β β β Intent matching, β ./bot/classifierβ β
β β β Symentic Processing β β β
β β β β β β
β β ββββββββββββ¬βββββββββββ β β
β β β Intent, Entities β β
β β ββββββββββββΌβββββββββββ β β
β β β β β β
β β β Intent Actor β ./bot/actions β β
β β β β β β
β βββ΄βββββββββββ¬βββββββββββ΄ββββββββββββββββββββ β
β β Graph Query β
β ββββββββββββΌβββββββββββ β
β β β β
β β Graph Database β Nebula Graph β
β β β β
β βββββββββββββββββββββββ β
β β
β β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
.
βββ README.md
βββ src
βΒ Β βββ siwi # Siwi-API Backend
βΒ Β βΒ Β βββ app # Web Server, take HTTP requests and calls Bot API
βΒ Β βΒ Β βββ bot # Bot API
βΒ Β βΒ Β βββ actions # Take Intent, Slots, Query Knowledge Graph here
βΒ Β βΒ Β βββ bot # Entrypoint of the Bot API
βΒ Β βΒ Β βββ classifier # Symentic Parsing, Intent Matching, Slot Filling
βΒ Β βΒ Β βββ test # Example Data Source as equivalent/mocked module
βΒ Β βββ siwi_frontend # Browser End
βΒ Β βββ README.md
βΒ Β βββ package.json
βΒ Β βββ src
βΒ Β Β Β βββ App.vue # Listening to user and pass Questions to Siwi-API
βΒ Β Β Β βββ main.js
βββ wsgi.py
The backend relis on the Nebula Graph, an Open Source Distributed Graph Database.
Install Nebula Graph in oneliner:
curl -fsSL nebula-up.siwei.io/install.sh | bash
Load the basketballplayer dataset.
~/.nebula-up/console.sh
nebula-console -addr graphd -port 9669 -user root -p nebula -e ":play basketballplayer"
Install and run.
# Install siwi backend
python3 -m build
# Configure Nebula Graph Endpoint
export NG_ENDPOINTS=127.0.0.1:9669
# Run Backend API server
gunicorn --bind :5000 wsgi --workers 1 --threads 1 --timeout 60
For OpenFunction/ KNative
docker build -t weygu/siwi-api .
docker run --rm --name siwi-api \
--env=PORT=5000 \
--env=NG_ENDPOINTS=127.0.0.1:9669 \
--net=host \
weygu/siwi-api
Try it out Web API:
$ curl -s --header "Content-Type: application/json" \
--request POST \
--data '{"question": "What is the relationship between Yao Ming and Lakers?"}' \
http://192.168.8.128:5000/query | jq
{
"answer": "There are at least 23 relations between Yao Ming and Lakers, one relation path is: Yao Ming follows Shaquille O'Neal serves Lakers."
}
Call Bot Python API:
from nebula3.gclient.net import ConnectionPool
from nebula3.Config import Config
# define a config
config = Config()
config.max_connection_pool_size = 10
# init connection pool
connection_pool = ConnectionPool()
# if the given servers are ok, return true, else return false
ok = connection_pool.init([('127.0.0.1', 9669)], config)
# import siwi bot
from siwi.bot import bot
# instantiate a bot
b = bot.SiwiBot(connection_pool)
# make the question query
b.query("Which team had Jonathon Simmons served?")
Then a response will be like this:
In [4]: b.query("Which team had Jonathon Simmons serv
...: ed?")
[DEBUG] ServeAction intent: {'entities': {'Jonathon Simmons': 'player'}, 'intents': ('serve',)}
[DEBUG] query for RelationshipAction:
USE basketballplayer;
MATCH p=(v)-[e:serve*1]->(v1) WHERE id(v) == "player112"
RETURN p LIMIT 100;
[2021-07-02 02:59:36,392]:Get connection to ('127.0.0.1', 9669)
Out[4]: 'Jonathon Simmons had served 3 teams. Spurs from 2015 to 2015; 76ers from 2019 to 2019; Magic from 2017 to 2017; '
Referring to siwi_frontend
βββββββββββββββββββββββββββββββ
β kind: Ingress β βββββββββββββββββββββ
β path: / β β Pod β
β -> siwi-frontend βββββΌββββββ€ siwi-frontend β
β β β β
β β βββββββββββββββββββββ
β β
β path: /query β βββββββββββββββββββββββββββββββββββββ
β -> siwi-api βββββΌββββββ€ KNative Service β
β KNative Serving β β serving-xxxx β
β β β β
β β β apiVersion: serving.knative.dev/v1β
β β β kind: Service β
βββββββββββββββββββββββββββββββ βββββββββββ¬ββββββββββββββββββββββββββ
β
ββββββββββββββ
β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
βapiVersion: core.openfunction.io/v1alpha1 β β
βkind: Function β β
βspec: β β
β version: "v1.0.0" β β
β image: "weygu/siwi-api:latest" β β
β imageCredentials: β β
β name: push-secret β β
β port: 8080 β β
β build: β β
β builder: openfunction/builder:v1 β β
β env: β β
β FUNC_NAME: "siwi_api" β β
β FUNC_TYPE: "http" β β
β FUNC_SRC: "main.py" β β
β srcRepo: β β
β url: "https://github.com/wey-gu/nebula-siwi.git" β β
β sourceSubPath: "src" β β
β serving: β β
β runtime: Knative ββββββββββββββββββββββββββββββββββΌβββ
β params: β
β NG_ENDPOINTS: "NEBULA_GRAPH_ENDPOINT" β
β template: β β
β containers: β β
β - name: function β β
β imagePullPolicy: Always β β
βββββββββββββββββββββββββββββββββββββββββΌββββββββββββββββ
β
ββββββββββββ
β
ββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββ
βapiVersion:lapps.nebula-graph.io/v1alpha1 β
βkind: NebulaCluster β
βspec: β
β graphd: β
β config: β
β system_memory_high_watermark_ratio: "1.0" β
β image: vesoft/nebula-graphd β
β replicas: 1 β
β... β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Assumed we have a k8s with OpenFunctions installed
Install a Nebula Graph with kubesphere-all-in-one nebula installer on KubeSphere:
curl -sL nebula-kind.siwei.io/install-ks-1.sh | bash
Get Nebula Graph NodePort:
NEBULA_GRAPH_ENDPOINT=$(kubectl get svc nebula-graphd-svc-nodeport -o yaml -o jsonpath='{.spec.clusterIP}:{.spec.ports[0].port}')
echo $NEBULA_GRAPH_ENDPOINT
Load Dataset into the nebula cluster:
wget https://docs.nebula-graph.io/2.0/basketballplayer-2.X.ngql
~/.nebula-kind/bin/console -u root -p password --address=<nebula-graphd-svc-nodeport> --port=32669 -f basketballplayer-2.X.ngql
Create the siwi-api powered by Openfunction:
cat siwi-api-function.yaml | sed "s/NEBULA_GRAPH_ENDPOINT/$NEBULA_GRAPH_ENDPOINT/g" | kubectl apply -f -
Get the function nebula-siwi and the KNative Service:
kubectl get functions nebula-siwi
FUNCTION=$(kubectl get functions nebula-siwi -o go-template='{{.status.serving.resourceRef}}')
kubectl get ksvc -l openfunction.io/serving=$FUNCTION
KSVC=$(kubectl get ksvc -l openfunction.io/serving=$FUNCTION -o=jsonpath='{.items[0].metadata.name}')
kubectl get revision -l serving.knative.dev/service=$KSVC
REVISION=$(kubectl get revision -l serving.knative.dev/service=$KSVC -o=jsonpath='{.items[0].metadata.name}')
echo $REVISION
Verify the function worked fine:
curl -s --header "Content-Type: application/json" \
--request POST \
--data '{"question": "What is the relationship between Yao Ming and Lakers ?"}' \
$(kubectl get ksvc -l openfunction.io/serving=$FUNCTION -o=jsonpath='{.items[0].status.url}')/query
Create the siwi-app resources on K8s:
cat siwi-app.yaml | sed "s/REVISION/$REVISION/g" | kubectl apply -f -
Verify the function worked fine through the ingress:
Here nodeport with http port 31059 was used as ingress controller endpoint.
curl -s --header "Content-Type: application/json" \
--request POST \
--data '{"question": "how does Tim Duncan and Lakers connected?"}' \
demo-siwi.local:31059/query
Verify the frontend:
curl $(kubectl get svc -l app=siwi -o=jsonpath='{.items[0].spec.clusterIP}')
Verify the frontend beind the ingress:
curl demo-siwi.local:31059
Get all resources in siwi-app:
kubectl get service,pod,ingress,function -l app=siwi
And it should be something like this:
[root@wey nebula-siwi]# kubectl get service,pod,ingress,function -l app=siwi
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/siwi-frontend-file ClusterIP 10.233.60.81 <none> 80/TCP 64m
NAME READY STATUS RESTARTS AGE
pod/siwi-frontend-file 1/1 Running 0 64m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/siwi-service <none> demo-siwi.local 80 59m
NAME BUILDSTATE SERVINGSTATE BUILDER SERVING AGE
function.core.openfunction.io/nebula-siwi Succeeded Running builder-sbfz6 serving-vvjvl 26h
[root@wey nebula-siwi]# kubectl get service,pod,ingress,function -l app=siwi
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/siwi-frontend-file ClusterIP 10.233.60.81 <none> 80/TCP 65m
NAME READY STATUS RESTARTS AGE
pod/siwi-frontend-file 1/1 Running 0 65m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/siwi-service <none> demo-siwi.local 80 59m
NAME BUILDSTATE SERVINGSTATE BUILDER SERVING AGE
function.core.openfunction.io/nebula-siwi Succeeded Running builder-sbfz6 serving-vvjvl 26h
docker build -t weygu/siwi-frontend . -f Dockerfile.froentend
docker push weygu/siwi-frontend
FAQs
A PoC of Dialog System With Graph Database Backed Knowledge Graph.
We found that siwi demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago.Β It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
CISA denies CVE funding issues amid backlash over a new CVE foundation formed by board members, raising concerns about transparency and program governance.
Product
Weβre excited to announce a powerful new capability in Socket: historical data and enhanced analytics.
Product
Module Reachability filters out unreachable CVEs so you can focus on vulnerabilities that actually matter to your application.