Theengs to Try Out - BLE Sensors, Theengs Gateway, and Home Assistant

I know I haven’t introduced my home lab or smart home setup yet. But I recently bought a few new sensors and wanted to share some new theengs I set up.

See what I did there? 😉

Yes, I installed Theengs Gateway on a Raspberry Pi 4B so that it would act as a gateway between my bluetooth low energy (BLE) sensors and my smart home’s MQTT Broker. MQTT is a lightweight messaging protocol which is often used in IoT and smart home integrations. It does this via OpenMQTTGateway, and Theengs is like a wrapper around it so that it is easily stood up on a Raspberry Pi or other devices.

Sensors Galore

I feel like I have an addiction to making our house “smart”. Really at the end of the day I want to supplement what we already have with additional functionality to improve our every day lives, and if I can add some cool extra stuff along the way then bonus points to me! Sensors are a key component to any smart home. There is a brutal heat wave happening here in Georgia and much of the United States, and I want to keep track of the temperature and humidity in my daughter’s room and the room my son will be sleeping in (hopefully soon 🤞). I’ve tried a few different temperature/humidity sensors, but I wanted to try something new. I ended up buying a few SwitchBot Indoor/Outdoor Thermo Hygrometers and I needed a way to integrate them into Home Assistant.

Home Assistant Integration

First, these thermo-hygrometers operate over BLE. I was able to pair these sensors with my phone via the SwitchBot app very easily, however I wanted to be able to view and use the data in my Home Assistant instance.

Enter Theengs Gateway.

Theengs Gateway is a BLE to MQTT bridge. It receives nearby device BLE packets, decodes them, and sends the data to a MQTT broker. In this case, I am running the Home Assistant MQTT Integration and the Mosquitto MQTT Add-on Home Assistant add-on, which installs the Eclipse Mosquitto MQTT broker. It is very simple to set up given the links provided. I won’t be covering that today, but I will give an overview in another series of articles more tailored to setting up Home Assistant.

Hardware Choice

Like many of you, I have a few extra Raspberry Pi’s laying around after years of collecting them. I have an original Pi, a Pi 3, some Zero W’s, and a couple Pi 4 Model B’s. My Home Assistant instance is running as a VM on a thin-client PC. I’ll get more into details in another article, but hey, as a young family man doing this as a hobby I don’t need to much, so the thin-client PC works great. However, I wanted to be able to reliably receive data from BLE devices on the other side of the house or even outside. Instead of running this gateway directly inside of Home Assistant and limiting my range, I decided to install Theengs Gateway on a Raspberry Pi 4 Model B so that I could put it anywhere within my wifi range and more reliably work with BLE devices.

I initially tried a Pi Zero W, but I had a few issues, primarily my frustration in waiting 😆. The Pi Zero W has some great applications, but I decided to go with the Pi 4 in this case to speed things up and have greater flexibility in the future if I decided to add additional protocol gateways via the USB ports. The Pi 4 has wifi and bluetooth built in, and was a great option to get things up and running. However, Theengs can run on a wide variety of hardware described on their website, including the popular ESP32.

What We’re Setting Up Today

Here’s a diagram of what we’re setting up today using the Mermaid diagramming tool:

  %%{init: { "themeCSS": ".edgeLabel { padding: 10px; }" } }%%
  graph TD
  subgraph SGTitle["`**Theengs BLE to MQTT Gateway**`"]
  end

  A[Sensor] -- Data over BLE --> D[Theengs Gateway]
  B[Phone] -- Data over BLE --> D
  C[Device] -- Data over BLE --> D
  D -- MQTT Publish --> E[MQTT Broker]
  F[Home Assistant] -- MQTT Subscribe --> E
    

I know it’s not proper UML, but it gets the job done and it’s something I’ll improve upon the more I add to these articles. Now let’s break this down, and really it is pretty simple. This is a sample diagram of how things are set up, in particular in my house. On the left side you have multiple BLE devices emitting data. Theengs Gateway, which is running on the Raspberry Pi 4, receives and decodes these BLE signals even without pairing (since it’s BLE). Theengs Gateway then publishes the data via MQTT to a particular topic on the MQTT Broker running on my Proxmox machine. Finally, Home Assistant, which is running on my Proxmox machine as well, subscribes to the same topic to read the data so I can use it within Home Assistant. Not too shabby!

Raspberry Pi Setup

Ok, the first thing I did as usual is use an imaging tool, in this case the official Raspberry Pi Imager, to flash the lite version Raspberry Pi OS since I will be using this headless without any monitors. You get the option to set wifi, initial user, ssh options, and more via the imager.

Boom, first roadblock.

I spent hours between the Pi Zero W and the Pi 4 flashing and reflashing, trying to troubleshoot why the Pi’s weren’t connecting to my wifi network on first boot. Then I found out why, and let me tell you to save you some headaches…

I think this was the first time I had one of my Pis connect to the network wirelessly since I upgrade network equipment to Unifi several months ago. It turns out that the official imaging tool does not support the changes needed to connect to a mixed security network, AND you cannot set it manually in the wpa_supplicant.conf file in Bookworm or newer as that functionality was deprecated (https://forums.raspberrypi.com/viewtopic.php?t=357623).

From what I read on a few forums, it all depends on what chipset your Pi has. If you run into this issue, the easiest solution, if you can, is to set the security on your wifi network to only WPA2 instead of WPA3/WPA2 mixed, and even better if you can set it to use only 2.4 GHz and not 5 GHz. Your other solution is to enable SSH access in the imager before flashing and hard wire the Pi for the first boot. Once you are able to SSH into the Pi, run the following command:

sudo raspi-config

You can update your network settings within this tool as well as play around with other settings.

Whew…

Installing Theengs Gateway

Let’s get to the good stuff.

As usual, I like linking to the original and official docs for anyone that wants to see them: Installing Theengs Gateway

There are multiple routes to install Theengs Gateway on your device. In my case, I went for the pip package in a Python virtual environment route. I tried the Snap package route on the Pi Zero and only led to frustration (probably because of the Pi Zero itself), so this seemed like a good route to try.

Python3 was already installed on my Pi, but I needed to install pip and the venv module. To do this, run the following commands:

sudo apt update
sudo apt install python3-pip python3-venv

Once that completes, you want to create your virtual environment and activate it. Running the following commands creates a virtual environment named theengs and activates it. Just a heads up - I did not run these with sudo and everything worked out great. It creates the virtual environment inside of your user folder and I did not run into any permission issues when actually running the gateway, even as a service.

python3 -m venv theengs
source theengs/bin/activate

You know you have activated the virtual environment when the command prompt is preceded by the virtual environment name, in this case (theengs). Now you can finally install the Theengs Gateway package by running the following command:

pip install TheengsGateway

After the installation, you can view the help documentation by running:

python3 -m TheengsGateway -h

Congrats, you have it installed! If you want to exit now and come back, or run a TheengGateway command in another shell session, you’ll need to make sure you first activate the same virtual environment by running

source theengs/bin/activate

If you used a different virtual environment name, substitute it out for theengs in that example. Now to configure the gateway for your first run.

First Run

Again, here is the official docs for reference.

Before we launch the gateway, if you don’t have a MQTT client that helps visualize the activity on your broker, you’ll want to install something. I installed MQTT Explorer on my Windows machine. You just need to enter in your broker’s settings, connect, and you’ll be able to watch Theengs Gateway publish data in real time.

To launch Theengs Gateway following this installation with the minimal setup, run the following command inside of your virtual environment:

python -m TheengsGateway -H "<mqtt_broker_host_ip>" -u "username" -p "password"

Make sure you put in your broker’s IP address, username, and password accordingly. And that’s it! You will be able to see all kinds of near by BLE devices reported on your MQTT broker. My suggestion is to stop it after you confirm it’s working so we can set it up as a service and we can set a whitelist of devices we want to track.

Running as a Service

I wasn’t a fan of just running it from the command line, so thankfully it was pretty simple to create a service that would automatically restart on reboot. Modify the template below for the TheengsGateway.service file with the following updates:

1. Replace the value in the ExecStart line with your original command
2. Replace pi for the value in the User line to your local username (not root)
3. Finally, if you’ve installed Theengs Gateway in a virtual python environment like we did in this article, change the ExecStart binary reference from /user/bin/python3 to /home/<user>/theengs/bin/python3

[Unit]
Description=Theengs Gateway
Requires=bluetooth.target network-online.target

[Service]
Restart=always
Type=simple
User=pi
ExecStart=/usr/bin/python3 -m TheengsGateway -H mqtt_broker_host_ip -u username -p password

[Install]
WantedBy=multi-user.target

We’re now going to install the service and enable it to start on boot. After we start it the first time, the service will make a configuration file we’ll want to get familiar with and possibly update. Run the following commands to enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now TheengsGateway.service

Check the status to makes sure it is running:

systemctl status TheengsGateway

You can check the logs to look for any issues if it is not:

journalctl -xe

And finally stop the service so we can take a look at the configuration it made on the first run:

sudo systemctl stop TheengsGateway.service

Configuration

The first time I connected Theengs Gateway to my MQTT broker, it automatically published data to the auto-discovery topic that Home Assistant subscribes to for EVERY BLE device it picked up. To say I was overwhelmed was an understatement. Fortunately, I was able to stop it as soon as I saw 30 different devices show up in Home Assistant that I didn’t own, and it didn’t take too long to clean things up.

I did some digging and found the device whitelist setting in the Theengs Gateway configuration file. In case you didn’t read the official docs and followed the approach I did, the gateway stores its configuration in a file theengsgw.conf in your home directory. So after the systemd service has started successfully, you may remove all options from the ExecStart line in the systemd service file, as the gateway reads the configuration from that file.

Take a look at the configuration file at ~/theengsgw.conf, it should look similar to the following, but you’ll have your host IP, gateway ID, MQTT user and password in it:

{
    "adapter": "",
    "bindkeys": {},
    "blacklist": [],
    "ble": 1,
    "ble_scan_time": 7,
    "ble_time_between_scans": 5,
    "ca_certs": null,
    "discovery": 1,
    "discovery_device_name": "TheengsGateway",
    "discovery_filter": [
        "IBEACON",
    ],
    "discovery_topic": "/home/TheengsGateway/BTtoMQTT",
    "enable_multi_gtw_sync": 1,
    "enable_tls": 0,
    "enable_websocket": 0,
    "gateway_id": "<your-gateway-ip>",
    "general_presence": 0,
    "host": "<host ip address>",
    "identities": {},
    "ignore_wblist": 0,
    "log_level": "INFO",
    "lwt_topic": "home/TheengsGateway/LWT",
    "pass": "<mqtt-password>",
    "port": 1883,
    "presence": 0,
    "presence_topic": "home/TheengsGateway/presence",
    "publish_advdata": 0,
    "publish_all": 1,
    "publish_topic": "home/TheengsGateway/BTtoMQTT",
    "scanning_mode": "active",
    "subscribe_topic": "home/+/BTtoMQTT/undecoded",
    "time_format": 0,
    "time_sync": [],
    "tls_insecure": 0,
    "tracker_timeout": 120,
    "trackersync_topic": "home/internal/trackersync",
    "user": "<mqtt-user>",
    "whitelist": []
}

I may have modified one or two other things, but it should look similar.

Now my piece of advice is to get the bluetooth MAC address for devices you want to track and enter them in the whitelist array as strings. This will prevent unwanted data from coming in, but it can be useful to leave it empty if you want to discover what’s around you.

Now, let’s update the service file to run the correct command now that it will automatically look at this file. Remove any parameters from the ExecStart line in /etc/systemd/system/TheengsGateway.service so that it ends with -m TheengsGateway.

Finally, let’s restart the service by running

sudo systemctl start TheengsGateway.service

And we’re set!

Take a look at the rest of the Theengs Gateway configuration, play around with it, and modify as you want!

Don’t forget to take a look at your MQTT broker to see all the data flowing in! If you have the MQTT integration installed in Home Assistant with auto-discovery enabled and the default discovery prefix, you should start seeing devices show up automatically without needing to change the Theengs Gateway configuration.

Summary

Whew, that was a lengthy article, but hopefully helpful. To recap, we:

• Installed Theengs Gateway on a Pi 4
• Bridged BLE sensor data into Home Assistant via MQTT via Theengs Gateway
• Set up a systemd service to make it persistent
• Whitelisted sensors to reduce noise

Next Steps - CI/CD

That was fun and all getting useful data into Home Assistant to use with everything we have installed there, but if you’re like me, I want to tweak the Theengs Gateway configuration to see what it can do. And if you’re also like me, you don’t make backups before changing things, and you’ll lose track of your changes along the way 😟😆

In my next article, I’ll show you how I put the Theengs Gateway configuration in a git repository and set up a deployment pipeline. This way, I’m able to always track my changes, make those changes without actually connecting to the Raspberry Pi, restart the service to pick up the changes, all while integrating a secret manager outside of GitHub so we don’t have sensitive data stored in the config!

Thanks for reading—drop a comment or share if this helped!