Thingplus (Thing+) is Software as a Service (SaaS) for Internet of Things (IoT) application services. For service partners looking to deliver an IoT experience to their users, it provides not only the applications running on cloud infrastructure but also the embedded middleware and client applications, helping them launch services quickly and easily. Drawing on the experience of developing Thingplus, this article looks at what to consider when developing cloud-based application services.
About the Author
Youngsoo Lee yougsoo.lee@daliworks.net | As CTO of Daliworks, he develops the Thing+ service together with Daliworks’ full-stack developers. He has product development experience in the Java VM and embedded fields at Aromasoft, Sun Microsystems, and Oracle.
Compiled by | Reporter Jae-seok Yoo yoojs@imaso.co.kr
Considerations for Scalability and High Availability
Thingplus has a scalable architecture: all services can run on a single virtual machine (VM) to set up a development environment, or run across dozens of VMs or more for commercial service. It also maintains high availability through failover. To satisfy scalability and high availability, the following factors must be considered even when adding one small feature.
– When one instance has a problem, is failover possible so that another instance can immediately take over the function?
– When the load gets heavy, can it be distributed so the problem is solved by adding more instances? Can it be sharded? Can a job queue be used?
– In what form (active/standby, active/active, master/slave) should multiple instances run?
Considerations for Improving Full-Stack Developer Productivity
With Thingplus, even adding a single simple feature frequently requires modifying many kinds of code: the embedded middleware for device connectivity, the cloud application, and the client applications. So that full-stack developers can add such features easily, we use the same development language, JavaScript, wherever possible and try to keep the same libraries and coding conventions. For example, the lodash (https://lodash.com/) library is used identically in the embedded middleware, on the server side, and in web browser client code.
Automation, Automation, and Automation
Operating Thingplus requires managing multiple servers and performing a variety of tasks, such as releases to roll out new versions of applications. The cloud must therefore be built so that these tasks can be automated. For Thingplus, we write our own automation code using node control, shell scripts, and similar tools. Going forward, we are also considering adopting tools such as ansible for automated deployment of server applications and instances.
In addition, to keep this complex system in a stable state at all times, we run the following automated tests.
– Unit tests: CI (continuous integration) runs on every source code commit, and unit tests are executed at that point
– Interactive user interface tests: We periodically run interactive UI tests of the user application.
– End-to-end tests: We continuously verify the entire flow in which data generated by IoT devices is stored on the server, rules are executed accordingly, and the results are recorded on the timeline.
A Cloud-Infrastructure-Independent Architecture and Minimizing Usage Costs
To minimize features that depend on any specific cloud infrastructure, we use container technologies such as docker so that Thingplus can run on Amazon Web Services (AWS), Azure, and private clouds. We also implement Thingplus features to operate at minimum cost by understanding exactly how cloud infrastructure usage is billed. At the same time, the user experience must not suffer, which always makes this a difficult area.
To apply the considerations described above, we developed ‘zkHelper’, which makes ZooKeeper easier to use, and we make use of ‘nightwatch.js’, a browser test automation tool.
zkHelper – A Distributed Processing Helper Built on ZooKeeper
In the Thingplus service, hundreds of servers (in this article, not physical servers but server-side applications that each handle a unit of work) cooperate to divide up and carry out complex work.
In distributed environments, ZooKeeper is widely used to manage many servers and assign their roles. There are other alternatives, but it is the most popular tool for playing the coordinator role in a distributed environment. ZooKeeper has the following characteristics.
– It is managed in units of znodes.
– A znode has a directory structure (path) similar to a file system.
– A znode must have a parent node, and the path of the root node is ‘/’.
– A znode can store data.
– A znode can be notified of changes to a path or to the stored data.
– A znode can be set as an Ephemeral node, which is automatically deleted when the connection between ZooKeeper and the client is lost.
The Thingplus development team used these ZooKeeper characteristics to release zkHelper as open source software so that it can be used more easily in distributed environments. Using it, let us look at how znodes change when a master is actually elected.
When the Node#1 server application initializes zkHelper with the options in <List 1>, nodes are created in ZooKeeper as shown in <List 2>.
When the node#2 server application additionally starts, it receives ticket number 2 as /myapps/votes/n_00002. However, since node#1 holds ticket number 1 (the earliest ticket), it retains its master status.
<List 1> zkHelper initialization options
{
basePath: 'myapps'
node: node#1:1234 // {hostname}:{port}
}
<List 2> Nodes created in ZooKeeper
/myapps
<List 3> node#1 retains master status
/myapps
Likewise, if the node#3 and node#4 server applications start as well, child nodes will be created under the /myapps/nodes node and the /myapps/votes node, and the master will not change.
If a failure occurs and the master node#1 server application loses its connection to ZooKeeper, that node disappears (since it is registered as an Ephemeral node). This triggers a new master election, at which point all servers participating in the election restart. They then compete fairly, taking tickets in turn, and the server with the lowest ticket number becomes the master.
<List 4> shows the znode state when node#3 has become the master.
<List 4> node#3 as the new master
/myapps
Next, let us look at zkHelper’s rules for master election. Assume ZooKeeper’s time unit (tick) is 2 seconds and the threshold for judging that a server has disconnected (sessionTimeout) is 10 seconds. When a master election is triggered, all servers restart in order to compete. If the restart was for simple maintenance purposes, the incumbent is favored: if the master comes back within 10 seconds (sessionTimeout), it keeps its master status. Occasionally, when connections drop due to poor network conditions between servers, master elections occur frequently. To minimize this, even if the network is temporarily disconnected, a retry is attempted after 5 seconds. The reason for choosing 5 seconds is that it is twice the ZooKeeper time unit (tick) with a 50% margin.
<List 5> shows how to use zkHelper. It is an example of participating in a master server election to decide master/slave in an environment running three ZooKeeper servers. There is also an example of monitoring in Observer mode without directly participating in the master election. It checks whether server applications are added or removed and whether the master changes.
<List 5> Participating in a master server election in an environment running three ZooKeeper servers
var zk = require('zkHelper'),
options = {
basePath: '/myapps';
configPath: '/myapps/config',
node: require('os').hostname(),
servers: ['zk0:2181', 'zk1:2181', 'zk2:2181'], // zk servers
clientOptons: {
sessionTimeout: 10000,
retries: 3
}
};
zk.init(options, function (err, zkClient) {
var appConfiguration;
if (zk.isMaster()) {
console.info('i am master')
} else {
console.info('master', zk.getMaster() && zk.getMaster().master)
}
appConfiguration = zk.getConfig();
// do something
});
<List 6> Monitoring a master election as an observer
var zk = require('zkHelper'),
_ = require('lodash'),
options = {
servers: ['zk0:2181', 'zk1:2181', 'zk2:2181'], // zk servers
clientOptons: {
sessionTimeout: 10000,
retries: 3
},
observerOnly: true
};
zk.init(options, function (err, zkClient) {
var observer = new Observer('/otherApp');
observer.on('children', _.debounce(function (path, newVal, diff) {
consol.info('Master=%j', observer.getMaster());
logger.info('children:[%s] add=[%s] del=[%s]', newVal, diff.added, diff.deleted);
}, 500));
observer.on('data', function (path, newVal, oldVal) {
if (oldVal) {
logger.info('master change:' + path);
}
logger.info('Master=%j', observer.getMaster());
logger.info('[%s] data: %j
Nightwatch.js – Web Application Test Automation
Nightwatch is an automation tool that tests in a browser under the same conditions as a real user’s environment, rather than running unit tests on web development code.
In web development, there are good tools and a variety of methods for unit testing on both the server and the client. The Thingplus service uses them on both the server and the client as well, but unit tests alone can miss errors that may occur in the user’s web environment. To prevent this, end-to-end tests that replicate the user’s environment are needed. Nightwatch is a useful tool that lets you create and test a variety of user environments with simple syntax and configuration.
Nightwatch Architecture
Nightwatch is a node.js-based automation tool built on Selenium, an open source project that provides a browser-based testing framework. Selenium runs several open source projects for web testing, and among them Nightwatch uses Selenium Webdriver to control the browser by communicating with the Selenium server.
Setting Up Nightwatch
First install node.js, then install Nightwatch. To use Nightwatch across the entire system, use the -g option.
$ npm install nightwatch
The Selenium server is Java-based. Install Java JDK version 6 or higher. Once Java is installed, download the latest version from the Selenium download page. selenium-server-standalone-2.45.0.jar is the latest version. The downloaded jar file can be run with the following command.
$ java -jar ./selenium-server-standalone-2.45.0.jar
If Selenium prints a message saying it has started successfully and enters a waiting state, it is working properly.
Next, a browser driver is a plugin that supports WebDriver’s wire protocol for communication between the Selenium server and the browser. You must install the corresponding driver for each browser you want to test. The full list of supported browser drivers can be found under the third-party drivers section of the Selenium download page.
Running Nightwatch
Once the drivers are installed, you need to configure the environment required to run Nightwatch. You can write the configuration easily by referring to the nightwatch.json example file provided on the Nightwatch GitHub (github).
When the nightwatch.json file is complete, run Nightwatch as follows.
$ nightwatch –config ./nightwatch.json –env integration
The –config option specifies the path to nightwatch.json, and the –env option takes one of the entries from the test_settings list configured above. If the –env option is not specified, default is executed. You can check detailed information about the options with –help.
Writing Test Cases
Save the sample below as a file in the src_folders path configured in nightwatch.json and run Nightwatch; you will see the Selenium server launch the browser and run through the test case automatically.
<List 7> Sample test file
module.exports = {
'step one' : function (browser) {
browser
.url('http://www.google.com')
.waitForElementVisible('body', 1000)
.setValue('input[type=text]', 'nightwatch')
.waitForElementVisible('button[name=btnG]', 1000)
}
};
The “browser” argument of the “step one” function in the sample source is a Nightwatch Object. Through this Object, you call the APIs provided by Nightwatch and carry out the tests you need. At the end of a test, you must call end() to close the session with the Selenium server so that the next test can proceed.
Two methods are provided for specifying elements: CSS selector and XPath. The default setting is CSS selector, so to use XPath you can add “use_xpath”: true to test_settings in nightwatch.json, or switch inside the test case as follows.
<List 8> Changing test case code
this.demoTestGoogle = function (browser) {
browser
.useXpath() // every selector now must be xpath
.click("//tr[@data-recordid]/span[text()='Search Text']")
.useCss() // we're back to CSS now
.setValue('input[type=text]', 'nightwatch')
};
That covers setting up Nightwatch and writing a simple test file. Nightwatch is under continuous development, with stability improvements and new features being added. More details are available on the Nightwatch homepage.
INTERVIEW Youngsoo Lee, CTO of Daliworks
Q We are curious what role Daliworks’ Thingplus plays.
A In short, Thingplus is SaaS for IoT application services. A detailed description is available on the Daliworks website (www.daliworks.net).
Q What were the key priorities in developing Thingplus?
A First, it must make connecting a wide variety of IoT devices easy. Second, it must be independent of the Infrastructure as a Service (IaaS) layer: it must be able to run anywhere, whether on AWS, Azure, or a private cloud. Third, it must be scalable. Under load it can run on dozens of virtual machines, and all services can even run on a single virtual machine.
Q We are curious whether the Thingplus platform runs only in the cloud, or whether it can also be built on separate servers inside a factory.
A At present, our business focuses on cloud-hosted services, but for large-scale cases we also support private cloud operation.
by Microsoftware Reporter Jae-seok Yoo | yoojs@imaso.co.kr
