Tests

Bork can be tested with Apache JMeter regarding of its functional correctness and performance aspect. With running instances of Orion Context Broker and Bork the following tests can be passed through (they may run on the same computer).

Functional testing

To verify the functional correctness of Bork a Apache JMeter test file can be run that covers various cases. The easiest way to run that test is using the JMeter GUI and enabling the individual test cases one after another (they should be run with a short timeout in-between). These test cases are named T.1 - T.4 and are described in the following.

Configure: Global Variables:

  • Host and Port of Bork (HOST and PORT)
  • Host and Port of Orion Context Broker (ORION_HOST and ORION_PORT)

Directly testing Bork

Each of the following test case represents a request directly to the Bork instance. This simulates notifications send by Orion and ensures that Bork's HTTP endpoint can handle different sources of error.

Test case Description Expected outcome Passed
T.1.0 HTTP-Body not JSON compliant Parsing failed - HTTP status code 400 OK
T.1.1 data field of the notification is missing Parsing failed - HTTP status code 400 OK
T.1.2 Exceed timeout time for notification processing HTTP status code 500 or 504 OK
T.1.3 Mandatory attribute of one or more entities is missing sequenceNumber is a required property - HTTP status code 400 OK
T.1.4 Additional attribute of a Heartbeat entity that is not defined in the schema additional properties not permitted and no patternProperties specified - HTTP status code 400 OK
T.1.5 The type of an attribute of one or more entities is not correct according to the schema sequenceNumber is not of type number - HTTP status code 400 OK
T.1.6 Missing type property The entity has no 'type' attribute - HTTP status code 400 OK

Testing Bork over Orion Context Broker

Each of the following test case represents a sequence of requests to the Orion Context Broker instance simulating an Agent and to check the existence of entities Bork should maintain. The individual test cases check the functional correctness of Bork regarding Mode 1 and Mode 2, with and without service path, as well as just with data updates.

Configure: User Defined Variables:

  • Heartbeat interval to be used by the simulated Agent (NOTIFY_INTERVAL)
  • Heartbeat interval that the simulated Agent should change to (NEW_NOTIFY_INTERVAL)
  • A slack which is used as a surcharge to the waiting time to publish the Heartbeat earlier or to delay it (NOTIFY_INTERVAL_SLACK_MS)
  • ServicePath Bork was started/configured with (BORK_SERVICE_PATH)
  • ServicePath the Agent entity should be published in (AGENT_SERVICE_PATH)
  • ServicePath the Heartbeat entity should be published in (HEARTBEAT_SERVICE_PATH)
  • Detection time upper bound (TuD) the Agent dictates Bork (DETECTION_TIME_UPPER_BOUND)
  • TuD to be changed to and the Agent dictates Bork (NEW_DETECTION_TIME_UPPER_BOUND)
Test case Description Expected outcome Passed
T.2.0 Mode 1: check if Discovery entity is present → publish Agent entity → publish Heartbeat m0…m3 → publish m3 → simulate failure → publish m0 → change Heartbeat interval η MonitoredAgent entity reflects the availability changes OK
T.2.1 see T.2.0 with Fiware-ServicePath see T.2.0 OK
T.2.2 see T.2.0 with DataUpdate see T.2.0 OK
T.2.3 see T.2.2 with Fiware-ServicePath see T.2.2 OK
T.3.0 Mode 1: check if Discovery entity is present → publish Agent entity → publish Heartbeat m0…mn → simulate failure → publish Heartbeat m0…mn MonitoredAgent entity reflects the availability changes OK
T.4.0 Mode 2 from high TuD to low: check if Discovery entity is present → publish Agent entity → publish Heartbeat m0 → change Heartbeat interval η according to the HeartbeatConfig entity → publish Heartbeat m1…mn and adjust when the η of the HeartbeatConfig entity changes → change TuD → repeat process → simulate failure → repeat process MonitoredAgent entity reflects the availability changes OK
T.4.1 see T.4.0 from low TuD to high see T.4.0 OK

Note: If required, the Jmeter-plugins-manager must be installed.

Load testing

To messure the performance of Bork a Apache JMeter test file can be run that simulates n Agents each of which publishes m heartbeats. Run the test file without using the GUI:

  • Windows: ./jmeter.bat -n -t path/test/load-testing.jmx
  • Linux: ./jmeter.sh -n -t path/test/load-testing.jmx

If changes need to be made to the parameters of the test file, then the GUI can be used for this case:

Configure:

  • Global Variables:
    • Host and Port of Bork (HOST and PORT)
    • Host and Port of Orion Context Broker (ORION_HOST and ORION_PORT)
  • Performance - processing time:
    • The field Number of Threads represents the number of Agents to be simulated
  • User Defined Variables:
    • Heartbeat interval to be used by the simulated Agents (NOTIFY_INTERVAL)
    • Number of Heartbeats each simulated Agent should send (NUMBER_OF_HEARTBEATS)

If the round trip time (RTT) from JMeter (simulated Agents) to a consumer is to be measured, the corresponding MonitoredAgent entities must be subscribed. If JMeter and the subscribing Application are on the same computer the receive minus the send time is just the RTT. When these two are not on the same computer, Bork should publish that send time in its MonitoredAgent entity too, so that it can be easily used to calculate the RTT on the subscribing computer.

If the processing time by Bork for one heartbeat is to be measured, the arrival time of the notification must be included in its bodies json string in the PostEntryPoint - handler() method (like notificationJson["data"][0]["borkReceivedTime"] = int(time() * 1000)). Then eg. in the UniqueAgent - addHeartbeat() method that time can be added to the to be send out MonitoredAgent entity (like self.monitoredAgent.current.borkReceivedTime = heartbeat.borkReceivedTime). To measure the end time of the processing, in the RequestSender - send() method the send time has must be included in the request that represents the MonitoredAgent entity (like request.body["borkSendTime"] = int(time() * 1000)). With this additional data a subscribing Application can calculate the processing time with send minus arrival time.

Evaluation

Because Bork communicates with the Orion Context Broker at every heartbeat, there will be more network communication than without it. The additional load on the network per hour, in Mode 1, can be estimated with approximately 3*(3600/interval). In Mode 1 with data update, approximately 2*(3600/interval) can be estimated. In Mode 2, at a change of the heartbeat interval all x heartbeats, with about (3600(3x+4))/(interval*x).

In a test environment, the various components were divided between three computers as follows:

  • The Orion Context Broker runs as a docker container on computer 1, which is equipped with an Intel i7-7700HQ 4-core CPU.
  • Computer 2 hosts Bork as a docker container and is equipped with an Intel i7-6600U 2-core CPU.
  • Computer 3, which is equipped with an Intel i5-7200U 2-core CPU, runs Apache JMeter, as well as a self-written Python program that subscribes to the context entities of the MonitoredAgent type and stores the test results in csv files.

Test setup

The computers are connected via a 100 Mbit/s Ethernet Local Area Network switch. Bork is configured with 4 threads and no authentication via Keyrock GE and Wilma GE is used.

Load Test

The load tests run by Apache JMeter on computer 3 simulates step by step 1 to 100 agents who also publish their heartbeat messages step by step with 1000 to 100ms heartbeat interval to the Orion Context Broker (running on computer 1). Bork processes these and the MonitoredAgent entities published on the context broker are subscribed by computer 3 that runs the Python program. Thus the RTT can be calculated, which is shown in the following diagram.

RTT Bork

The line diagram above shows that the RTT of one to 20 agents increases in most cases at every heartbeat interval. Subsequently, the lines and thus the RTT of the intervals 100ms, 250ms and 500ms tear out one after the other. At these times, the utilization of computer 1 and 2 was close to 100%, resulting in an extreme deterioration of the RTT.

Processing time Bork

The line diagram above shows the pure processing time of the heartbeats by Bork. In general, this is constant or falling for up to 35 agents. The processing time for the heartbeat intervals 100ms and 250ms increases significantly from this point, again due to the heavy load of computer 1 and 2. The falling trend can be explained by the constant utilization of the thread pools, whose workers do not have to wait for a signal from the thread pool before they can perform tasks.

Generally it can be concluded that Bork has to process a heartbeat within the used interval from the Agent, to function without running into a backlog. With the assumption that every Agent uses the same heartbeat interval, the inequality (processingTimeAgents)/Threads <= interval* must be complied.

CPU Time

To determine the additional CPU time required running Bork, two identical load tests with and without Bork were performed on computer 3 (Intel i5-7200U 2-core CPU). The load tests are based on 20 Agents updating measurement data in an interval of 500ms. The measurements were made with portainer.

CPU time only Orion

Initially, measurements were only taken with the Orion Context Broker and a connected MongoDB instance, each as a docker container, see Figure above. It can be seen that MongoDB requires more CPU time in this load test with an average value of 9.55% than the Orion Context Broker with an average value of 7.68%.

CPU time Orion and Bork

Afterwards the load test was repeated with Bork connected as Docker Container, see Figure above. This shows how the required CPU time of the Orion Context Broker increases to 16.6% on average. This additional load is triggered by the notifications sent to Bork and the updates of the MonitoredAgent entities. Bork has an average CPU time value of 38.67% for this load test. If the CPU time of the MongoDB instance is ignored, the additional load on required CPU time for this load test is 47.59% on average. The CPU and the number of its cores are an important factor for the vertical scaling. This test was only done on one specific computer with one specific configured bork, so this result should only be seen as a trend.

Thanks to the architecture of Bork using thread pools, it should scale with more powerful CPUs. Beside that multiple instances of Bork can run concurrent using one instance of a Orion Context Broker, making it horizontal scalable.