ENGINE Kibble project

Kibble probes devices with different protocols, stores status events in MongoDB, detects latency thresholds, and raises alerts.

Prerequisites

• Python 3.12+
• Docker Desktop (for the simulated environment and local MongoDB)
• sudo on Linux/macOS

Optional:

• MongoDB Compass - GUI for browsing the kibble database
• mongosh - MongoDB shell (included in the main simulator container)

Quick start (simulator)

This is the fastest way to try Kibble. Docker Desktop must be running before you begin.

git clone https://github.com/Geeoon/ENGINE-Kibble.git
cd ENGINE-Kibble

python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r src/Kibble/requirements.txt

cd simulator
./run.sh

run.sh opens a shell inside kibble-main-container. When you exit that shell, the whole stack is torn down Inside the main container shell, start the monitor:

cd /tmp
./start.sh

or run

Or run python3 main.py directly (equivalent inside the container).

On first run, main.py automatically seeds test devices (simulator-secondary-1 … 5, simulator-scpi-1 … 5, and a few others). No manual setup is required.

In another terminal on your host, open MongoDB Compass with:

mongodb://root:password@localhost:27017

In MongoDB Compass devices and configurations can also be manually added/removed. Open the kibble database and inspect timeseries_events for live probe results.

Main Monitor Installation

Use this path when you want to run the monitor on your host machine (outside Docker) against a local MongoDB container.

Linux

1. Clone the repository

   git clone https://github.com/Geeoon/ENGINE-Kibble.git
   cd ENGINE-Kibble

2. Create a virtual environment and install dependencies

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r src/Kibble/requirements.txt

3. Configure email alerts (optional)

4. • Create a Gmail email account with an app password
   • Modify the main.py args.sender_email and args.reciever_email
   • Modify the .env to set the EMAIL_PASSWD to the app passwork

5. Start MongoDB

   Either run the full simulator:

   cd simulator
   ./run.sh

   Or start only the database container:

   cd simulator
   mkdir -p db && sudo chmod 777 db
   docker compose up database -d

6. Run the monitor

   Inside the simulator container:

   cd /tmp
   ./start.sh

On the host machine:

src/main.py defaults to database.internal. For host-side runs, either add 127.0.0.1 database.internal to /etc/hosts, or change mongo_host in src/main.py to 'localhost'.

cd src
source ../.venv/bin/activate
./start.sh

start.sh loads ../.env and runs main.py with sudo when available (needed for ICMP/Scapy).

Press Ctrl+C to stop the monitor. Logs are written to kibble.log and kibble_status.log.

Windows

1. Clone the repository

   git clone https://github.com/Geeoon/ENGINE-Kibble.git
   cd ENGINE-Kibble

2. Create a virtual environment and install dependencies

   python -m venv .venv
   .venv\Scripts\activate
   pip install -r src/Kibble/requirements.txt

3. Configure email alerts (optional)

   • Create a Gmail email account with an app password
   • Modify the main.py args.sender_email and args.reciever_email
   • Modify the .env to set the EMAIL_PASSWD to the app passwork

4. Start MongoDB with Docker Desktop

   cd simulator
   mkdir db
   docker compose up database -d

   To run the full simulator without bash, build and start manually:

   docker compose build
   docker compose up --scale secondary=5 --scale scpi=5 -d
   docker exec -it kibble-main-container bash

5. Run the monitor

   Inside the simulator container:

   cd /tmp
   python3 main.py

   or

   cd /tmp
   ./start.sh

   (./start.sh and python3 main.py are equivalent inside the container)

   On the host machine:

   Change mongo_host in src/main.py to 'localhost', then:

   cd src
   ..\.venv\Scripts\activate
   python main.py

Simulated environment (for development)

The simulator/ directory runs a full test network in Docker: a monitoring host, MongoDB, secondary machines, and SCPI instrument simulators.

Start the stack

cd simulator
./run.sh          # starts 5 secondary + 5 SCPI devices (default)
./run.sh 3        # start 3 of each instead

run.sh will:

1. Build all container images
2. Start MongoDB and simulated devices
3. Open a shell inside kibble-main-container
4. Tear down the stack when you exit that shell (docker compose down -v)

To keep containers running in the background:

cd simulator
mkdir -p db && sudo chmod 777 db
docker compose build
docker compose up --scale secondary=5 --scale scpi=5 -d

The monitor connects to MongoDB at database.internal:27017, seeds test devices, and begins probing.

Configuration

Database

Kibble stores device configuration and monitoring events in MongoDB.

Setting	Simulator (inside Docker)	Host machine / Compass
Host	database.internal	localhost
Port	27017	27017
Username	root	root
Password	password	password
Database	kibble	kibble

Connection string for Compass or mongosh:

mongodb://root:password@localhost:27017

These credentials are simulator defaults for local development only.

Start the database container:

cd simulator && docker compose up database -d

Collections

Collection	Purpose
devices	Device identity (asset_tag, device_type_id)
device_types	Device type definitions and supported protocols
device_configurations	Which interfaces are active for a device
interface_configurations	IP, hostname, MAC, and other network details
timeseries_events	Monitoring events and latency readings

Device configuration tool (manage_devices.py)

Use src/manage_devices.py to add or remove monitored devices in MongoDB from the terminal. The monitor reads device configuration from the database on each scan cycle.

MongoDB must be running before using this tool.

How device data is stored:

What you set in the CLI	Where it goes in MongoDB
--asset-tag	devices.asset_tag
--type	devices.device_type_id → device_types
--ip, --hostname, --mac	interface_configurations and device_configurations

Workflow (simulator - devices are auto-seeded):

1. Start the simulator (or at least the MongoDB container)
2. Start the monitor - main.py seeds simulator devices on first run
3. Inspect results in Compass (timeseries_events)
4. Optionally add more devices with manage_devices.py while the monitor runs

Workflow (host / custom devices - manual setup):

1. Start MongoDB
2. Add devices with manage_devices.py
3. Verify in Compass (see below)
4. Start the monitor
5. Watch timeseries_events for probe results

Add a device

Each device needs a unique --asset-tag, a device type (--type or --type-name with --protocols), and at least one of --ip or --hostname.

--type	Protocols
icmp	ICMP
scpi	SCPI
snmp	SNMP
daemon	Daemon

Examples:

# Ping a device by IP
python manage_devices.py add --asset-tag 1001 --type icmp --ip 192.168.1.10

# Ping a device by hostname
python manage_devices.py add --asset-tag 1002 --type icmp --hostname my-server.local

# Add an SCPI instrument in the simulator (asset 3001 is auto-seeded; use a new tag to add another)
python manage_devices.py add --asset-tag 9002 --type scpi --hostname simulator-scpi-1

# Custom device type
python manage_devices.py add --asset-tag 3001 --type-name "custom probe" --protocols ICMP --ip 10.0.0.5

Inside the main simulator container (open a second shell with docker exec -it kibble-main-container bash while the monitor runs):

cd /tmp
python manage_devices.py --mongo-host database.internal add \
  --asset-tag 9001 --type icmp --hostname simulator-secondary-1

Re-running add with the same --asset-tag updates the device. A new configuration snapshot is recorded only when IP, hostname, or MAC changes.

Remove a device

python manage_devices.py remove --asset-tag 1001

This deletes the device and its associated configuration records. Historical timeseries_events are not deleted.

CLI reference

usage: manage_devices.py [-h] [--mongo-host MONGO_HOST] [--mongo-port MONGO_PORT]
                         {add,remove} ...

options:
  --mongo-host MONGO_HOST   default: localhost (use database.internal in Docker)
  --mongo-port MONGO_PORT   default: 27017

add:
  --asset-tag ASSET_TAG     required
  --type {daemon,icmp,scpi,snmp}
  --type-name TYPE_NAME     requires --protocols
  --protocols PROTOCOLS [PROTOCOLS ...]
  --ip IP
  --hostname HOSTNAME
  --mac MAC

remove:
  --asset-tag ASSET_TAG     required

Command-Line Arguments

usage: main.py [-h] [--low-thresh LOW_THRESH] [--medium-thresh MEDIUM_THRESH] [--high-thresh HIGH_THRESH] [--monitor-id MONITOR_ID] [--community-string COMMUNITY_STRING] [--device-timeout DEVICE_TIMEOUT] [--scan-period SCAN_PERIOD] [--threads THREADS]
               [--sender-email SENDER_EMAIL] [--receiver-email RECEIVER_EMAIL] [--mongo-host MONGO_HOST] [--mongo-port MONGO_PORT] [--mongo-user MONGO_USER] [--mongo-pass MONGO_PASS]

options:
  -h, --help            show this help message and exit
  --low-thresh LOW_THRESH
  --medium-thresh MEDIUM_THRESH
  --high-thresh HIGH_THRESH
  --monitor-id MONITOR_ID
                        Unique integer ID for this monitoring node (lowest ID wins leader election)
  --community-string COMMUNITY_STRING
                        Community string for SNMP monitor
  --device-timeout DEVICE_TIMEOUT
                        Timeout for the device
  --scan-period SCAN_PERIOD
                        How often to scan the network. Should be at least double the device timeout
  --threads THREADS     The number of threads to launch to do simultaneous device scans. Should scale with the number of devices.
  --sender-email SENDER_EMAIL
                        The email account to send alerts from
  --receiver-email RECEIVER_EMAIL
                        The email accoutn to send alerts to
  --mongo-host MONGO_HOST
                        The MongoDB hostname
  --mongo-port MONGO_PORT
                        The MongoDB port
  --mongo-user MONGO_USER
                        The MongoDB username
  --mongo-pass MONGO_PASS
                        The MongoDB password

MongoDB Compass

MongoDB Compass provides a GUI for inspecting the kibble database. Connect:

1. Install Compass from the MongoDB download page.

2. Start the database container: cd simulator && docker compose up database -d

3. Paste this connection string:

   mongodb://root:password@localhost:27017
   

4. Open the kibble database.

If Compass asks for an authentication database, use admin.

Custom Daemon Installation

Linux

• Copy edge_node directory onto device to be monitored

cd edge_node/linux/
./install.sh

Windows

• Copy edge_node directory onto device to be monitored
• Open Powershell as administrator

cd edge_node/windows/
./install.ps1