Issues in recoverability & reliability of communication between Gateway & Thingsboard CE server

[ankushdineshrana]

I have been trying to setup Thingsboard CE (latest) and Thingsboard Gateway. I have tried various installation methods but issue remains same.

Let me share details about the issue below:

1. I have installed Thingsboard Gateway & Thingsboard CE Server on EC2 Ubuntu (version 20.04) on two separate machines. Lets call them Machine-1 (Gateway) & Machine-2 (Server) respectively.

2. I have registered one gateway on Thingsboard server and used docker compose file generated post gateway registration to start gateway docker container on EC2 machine.

Docker-Compose.yml for gateway

version: '3.4'
services:
  # ThingsBoard IoT Gateway Service Configuration
  tb-gateway:
    image: thingsboard/tb-gateway
    container_name: tb-gateway
    restart: always

    # Ports bindings - required by some connectors
    ports:
        - "5000:5000" # Comment if you don't use REST connector and change if you use another port
        # Uncomment and modify the following ports based on connector usage:
#        - "1052:1052" # BACnet connector
#        - "5026:5026" # Modbus TCP connector (Modbus Slave)
#        - "50000:50000/tcp" # Socket connector with type TCP
#        - "50000:50000/udp" # Socket connector with type UDP

    # Necessary mapping for Linux
    extra_hosts:
      - "host.docker.internal:host-gateway"

    # Environment variables
    environment:
      - host=XX.XX.XXX.XXX  # Masked for security purposes only
      - port=1883
      - accessToken=TbdhVMvx1hlvXQHQlcyf

    # Volumes bind
    volumes:
      - tb-gw-config:/thingsboard_gateway/config
      - tb-gw-logs:/thingsboard_gateway/logs
      - tb-gw-extensions:/thingsboard_gateway/extensions

# Volumes declaration for configurations, extensions and configuration
volumes:
  tb-gw-config:
    name: tb-gw-config
  tb-gw-logs:
    name: tb-gw-logs
  tb-gw-extensions:
    name: tb-gw-extensions

3. I have opened ports suggested by Thingsboard documentation on both machines. (Let me admit even I opened all ports at some point of time to cross-out any port related issues)

4. I have started a local broker on machine-1 using following docker command
   sudo docker run -it -p 1884:1884 thingsboard/tb-gw-mqtt-broker:latest
   The above command runs a MQTT broker and also starts sending data on topic : data/

5. Here is connector information that I have configured for the gateway via Thingsboard Dashboard

Connector Information

{
  "broker": {
    "name": "Demo Broker 100",
    "host": "host.docker.internal",
    "port": 1884,
    "clientId": "ThingsBoard_gateway_100",
    "version": 5,
    "maxMessageNumberPerWorker": 10,
    "maxNumberOfWorkers": 100,
    "sendDataOnlyOnChange": false,
    "security": {
      "type": "anonymous"
    }
  },
  "mapping": [
    {
      "topicFilter": "data/",
      "converter": {
        "type": "json",
        "deviceNameJsonExpression": "Demo Device 100",
        "deviceTypeJsonExpression": "default",
        "sendDataOnlyOnChange": false,
        "timeout": 60000,
        "attributes": [
          {
            "type": "integer",
            "key": "frequency",
            "value": "${frequency}"
          },
          {
            "type": "integer",
            "key": "power",
            "value": "${power}"
          }
        ],
        "timeseries": [
          {
            "type": "integer",
            "key": "temperature",
            "value": "${temperature}"
          },
          {
            "type": "integer",
            "key": "humidity",
            "value": "${humidity}"
          }
        ]
      }
    }
  ],
  "connectRequests": [
    {
      "topicFilter": "sensor/connect",
      "deviceNameJsonExpression": "${SerialNumber}"
    },
    {
      "topicFilter": "sensor/+/connect",
      "deviceNameTopicExpression": "(?<=sensor/)(.*?)(?=/connect)"
    }
  ],
  "disconnectRequests": [
    {
      "topicFilter": "sensor/disconnect",
      "deviceNameJsonExpression": "${SerialNumber}"
    },
    {
      "topicFilter": "sensor/+/disconnect",
      "deviceNameTopicExpression": "(?<=sensor/)(.*?)(?=/disconnect)"
    }
  ],
  "attributeRequests": [],
  "attributeUpdates": [],
  "serverSideRpc": [],
  "logLevel": "DEBUG",
  "name": "MQTT100",
  "id": "90d4cfba-e1d0-4608-a44b-dc5c1fd1f69f"
}

6. As soon as I save the configuration 9/10 times device doesn't come to Sync Status. This part looks very unreliable. Sometimes it just connects but most of the times I have to struggle like enable/disable it multiple times. Often it connects when I restart my gateway container on machine-1.

7. After few restarts and enable/disable magic, sometimes it connects. I can also see data coming in continuously.

8. At this point of time if I try to check the reliability & recoverability of the system, I intentionally stop the gateway container and start it after 1 or 2 mins. 9/10 times system doesn't recover itself. No telemetry starts coming it remains stuck at last message only.

9. One more weird scenario, if I disable the connector, telemetry still keeps coming and either doesn't stop at all or stops after a long time.

10. Moreover status of the connector doesn't seems reliable to me, somehow it doesn't actually reflect the real status as in what's happening inside currently.

Sample Image

11. I have also tried cloud platform for Thingsboard (free trial). I am able to see same behavior.

12. QUESTION: Is anything that I am missing here or do you face similar issues?

13. I have also tried same with OPC-UA connector, same behavior was observed.

14. Please let me know if I need to provide any other details here. Please help.

Thanks! Your help will be much appreciated!

[github-actions]

Hi @ankushdineshrana.
Thank you for your interest in ThingsBoard IoT Gateway.
Your issue was registered, please wait for response from engineer.

IOTGW-182

[imbeacon]

Hi @ankushdineshrana,

The issue may be related to rate limits on TB, could you try to use the latest version of the gateway and try to reproduce the issue?

[ankushdineshrana]

Hi @ankushdineshrana,

The issue may be related to rate limits on TB, could you try to use the latest version of the gateway and try to reproduce the issue?

Hey @imbeacon ,

Thanks for your reply. I have tried with latest gateway version (3.4.6). Here are my observations:

1. Gateway version 3.4.6 seems to be more stable and reliable, It reconnects (9/10) times without restarting gateway/Thingsboard-server.

2. One strange thing I have observed that, since I am testing these out in EC2 instances. IP Changes after restarts so when I config new IP in the docker-compose file of tb-gateway and start it, gives errors related to socket timeouts. I am assuming it somehow tries to connect with the older (non-existent) IP.

3. I have tried by deleting docker images and start the tb-gateway container repeatedly, at time this succeeds and some times doesn't. Shows inconsistent behavior.

Can you please suggest if am missing something here. Thanks in anticipation.