Starting with a golden circle and the question "Why?"
The BigBoards Fabric, a major milestone
April 15, 2016
Show all

How to create your own BigBoards app?

Apache Ignite In-Memory Data Fabric consists of a bunch of components collaborating as jigsaw puzzles: data grid, compute grid, service grid, streaming, hadoop acceleration, advanced clustering, file system, messaging, events and data structures

Using Apache Ignite as an example, you will learn how to assemble your own BigBoards app. Once your BigBoards app is defined on the BigBoards Hive, you deploy it to your BigBoards Hex by the click of 1 button!

Introduction to Apache Ignite

I'm the organiser of BigData.be, the Belgian user community on Big Data, NoSQL and scalable technologies. On Tuesday, November 8th, 2016 we organised our 39th meetup.

As 2nd presentation of the evening, Mathias Lavaert from Dataminded introduced Apache Ignite, and more specifically, its Data Grid and Compute Grid components.

Apache Ignite In-Memory Data Fabric is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies.

During Mathias' presentation, I browsed the Apache Ignite website and noticed they have docker images readily available.

Apache Ignite's download section refers to Docker and Cloud images

Diving a bit deeper in the Apache Ignite documentation on docker deployments, it was clear that Apache Ignite's services inside the application container are completely tuneable through environment variables and configuration files.

Creating a BigBoards app for Apache Ignite on the BigBoards platform should be a piece of cake!

In this tutorial, you'll roll your own app to deploy Apache Ignite on your BigBoards Hex.

Create your first BigBoards app

To define your first BigBoards app, head over to our central platform. Login to the platform ... or Signup, if you haven't already. It's free!

Login icon on the bigboards hive shows a small key as recognisable item

Use the key icon at the top right. As explained in our documentation for the Hive, you can use

  • an email/password combination on our platform, or
  • authenticate using your social profile on GitHub or Google account.

Once logged in, you land on our library of available BigBoards apps.

The BigBoards library gives access to all install packages available to you. Packages are presented as clickable tiles.

To get started creating your own BigBoards app, use the button with the + sign at the Bottom right. It slides open.

Next, click the Add Stack button.

The new stack dialog asks to give your install package a unique name

Give your BigBoards app a unique identifier.

The system will flag the field, if your id is already in use by another BigBoards app!

We name all our BigBoards apps with `bb-stack-` to separate it from the rest.

Choose your own naming convention for your BigBoards apps.
In the near future, we'll have a lot more to deploy for you: only technologies, but also datasets, tutorials and many more!

General info

The 2nd step is to complete the general information of your newly created BigBoards app:

The first thing to complete for a BigBoards app are the general info like name, description, logo, ...

  • name is just a short label that is shown on the card in the library etc.
  • description is a longer piece of text to explain about the technology stack that you are assembling.
  • logo url is for showing a recognizable image in the library and on the cluster consoles.
  • git url points to a version control repository where we'll store all files and artifacts to configure the technologies which we will be deploying. This repository must be publicly accesible as we clone the complete repository to the target cluster on deployment.
  • Architecture allows you to constrain your BigBoards app to the given CPU architectures. Your BigBoards app will only show up in the library panel of the clusters that adhere to your selected architectures. We currently support Intel and ARM. Whether your BigBoards app supports 1 or the other architecture, or all, totally depends on what you are deploying.
  • Firmware constrains the availability of your BigBoards app to the BigBoards firmware that is installed on the target cluster. Normally everybody should be on the latest level!
  • Visibility allows you to keep your BigBoards app private to you and your collaborators, or you can share with the whole community, i.e. public.
  • Collaborators is the list of users that are allowed to use, but also alter(!) your BigBoards app.

As you can see, setting up your own BigBoards app for Apache Ignite is very simple.

  1. For vanilla technologies, you can easily copy the description from the origin website.
  2. Create your own git configuration repository on GitHub, BitBucket or GitLab. Make sure it is public!
  3. You can have a logo pointing directly to the origin website such as http://ignite.apache.org/images/logo.png, but experience proves that URLs change.

Use our configuration repositories at https://github.com/bigboards/bb-stack-* as reference material.
Commit the logo for your BigBoards app to your configuration repository and use the image's raw URL.
Changes to fields are saved automatically while editing. There are no separate OK, Confirm or Save buttons. No confirmation dialogs either!

Containers

3rd, we have to setup what needs to be deployed by your new BigBoards app. The BigBoards infrastructure deploys software services as docker containers.

What is docker? explains how you can package any application into a standardised unit for software development:
Docker containers wrap a piece of software in a complete filesystem, that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment.

But, Docker not only makes technologies, processes and services portable. It also makes them erasable!

Application containers ensure that the base system of your cluster remains clean from foreign contaminations. That is all important for the stability of your system!

In that respect, Apache Ignite is a simple solution: it requires just a single process to be depoyed. 1 docker container!

The bottom rights shows a large button with the plus sign and above it the add container button showing a wireframe cube

Use the button with the + sign at bottom right. It slides open to the top

Next, click the Add Container button to setup a new container:

The panel to add a container shows all the information to configure a docker container: name, docker image, command to run, networking setup, ports to map, volumes to mount, etc

  • image points to the docker images that needs installing. Notice that you can add a tag (1.7.0 in this case) or omit it entirely to pull the latest image.
  • registry is used to find the requested docker image. Blank uses the official docker hub.
  • command allows you to overrule the command inside the docker container. Useful if a single docker container can be used for multiple processes.
  • networking allows you to chose how the container integrates with the host's network: bridged or host. Most of the time we simply use host.
  • port mappings allow you to reroute ports from the docker container to other ports of the host. This is only required with networking set to bridged.
  • environment variables allow you to pass information from the host environment into the container.
  • volume mappings mount directories from the host as directories inside the containers.
  • volume links, container links and advanced settings are all advanced settings, only used in exceptional cases.

Use a tag to define your docker image to prevent unexpected behaviour.
Add all ports, even with host networking, to explicitly express which ports get exposed.

For adding an item (like a port mapping) to a list, you have to explicitly save the changes per line. To remove or update an entry, use the drop-down menu via the `hamburger` button.

Groups

Groups are the deployment descriptor of the BigBoards infrastructure: it defines which containers need to end up on which nodes in the cluster.

As your BigBoards app for Apache Ignite only has 1 container, it also only needs 1 group.

To add a group, the bottom right holds an big icon with a plus sign, that slides open to the top. The middle icon when open is to add a new group. The button shows a hierarchy diagram.

Use the button with the + sign at the bottom right. It slides open to the top.

Next, click the New Group button.

A group needs a name, a placement expression and a list of containers to deploy

  • name defines the unique name of the group within the app
  • placement expression targets specific nodes within the target cluster. Some examples:
    • all includes all cluster nodes
    • !n1 excludes the 1st node
    • all:!n2 targets all but the second node
    • n3,n4 targets node 3 and 4
  • containers is the list of containers that have to be deployed to the targeted nodes.

The container name field is an auto-complete field for the containers defined by the BigBoards app. Use the Save button at the right to add the container to the list.

Views

Apache Ignite has no views!

Views are for another tutorial!

Deploy your BigBoards app for the 1st time

The next step is to test your BigBoards app. So, go ahead and deploy it to your cluster:

  1. Head over to your Hex's dashboard,
  2. uninstall any currently installed BigBoards app,
  3. go to the dashboard's library,
  4. install your new BigBoards app

Installing an application on a BigBoards cluster output a full installation log. The final lines show for each nodes completed and failed tasks

The install logs shows that no problems occurred.

Let's check the installation:

  1. Login to the first node via ssh.
  2. Run the command docker ps to list running containers.
bb@optimus-prime-n1:~$ docker ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
2da47fa259b7        index.docker.io/apacheignite/ignite:1.7.0   "/bin/sh -c $IGNITE_H"   19 minutes ago      Up 19 minutes                           apache-ignite

Run the command docker logs apache-ignite to output the log of Apache Ignite

bb@optimus-prime-n1:~$ docker logs apache-ignite
[15:07:17]    __________  ________________
[15:07:17]   /  _/ ___/ |/ /  _/_  __/ __/
[15:07:17]  _/ // (7 7    // /  / / / _/
[15:07:17] /___/\___/_/|_/___/ /_/ /___/
[15:07:17]
[15:07:17] ver. 1.7.0#20160801-sha1:383273e3
[15:07:17] 2016 Copyright(C) Apache Software Foundation
[15:07:17]
[15:07:17] Ignite documentation: http://ignite.apache.org
[15:07:17]
[15:07:17] Quiet mode.
[15:07:17]   ^-- Logging to file '/opt/ignite/apache-ignite-fabric-1.7.0-bin/work/log/ignite-f826da25.0.log'
[15:07:17]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[15:07:17]
[15:07:17] OS: Linux 3.19.0-23-generic amd64
[15:07:17] VM information: OpenJDK Runtime Environment 1.8.0_102-8u102-b14.1-1~bpo8+1-b14 Oracle Corporation OpenJDK 64-Bit Server VM 25.102-b14
[15:07:17] Configured plugins:
[15:07:17]   ^-- None
[15:07:17]
[15:07:17] Security status [authentication=off, tls/ssl=off]
[15:07:20] Topology snapshot [ver=5, servers=5, clients=0, CPUs=20, heap=5.0GB]
[15:07:20] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[15:07:20]
[15:07:20] Ignite node started OK (id=f826da25)
[15:07:20] Topology snapshot [ver=5, servers=5, clients=0, CPUs=20, heap=5.0GB]
[15:07:22] Topology snapshot [ver=6, servers=6, clients=0, CPUs=24, heap=6.0GB]

As you can see, the Apache Ignite containers start properly ... and keeps running. No continuous restarts!

Custom configuration

Apache Ignite's console output also shows that the Ignite grid auto-discovers our 6 nodes within the Hex. Auto-discovery via TCP/IP is the default for Apache Ignite clustering!

While auto-discovery is extremely easy, it is also very dangerous in enterprise environments. Suppose that 2 clusters install your new BigBoards app. They would immediately bond together to swarm into a huge Ignite grid!

That's not what you had in mind for your production cluster, when you started your test environment :'‑(

To disable auto-discorvery, add a custom configuration to your Git repository: /config/ignite/default-config.xml:

<?xml version="1.0" encoding="UTF-8"?>

<!--
    Ignite Spring configuration file to startup ignite.
-->
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd">

    <!--
        Optional description.
    -->
    <description>
        Spring file for ignite configuration with static IP configuration.
    </description>

    <!--
        Initialize property configurer so we can reference environment variables.
    -->
    <bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
        <property name="searchSystemEnvironment" value="true"/>
    </bean>

    <!--
        Core ignite configuration
    -->
    <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
        <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <!--
                        Ignition provides several options for automatic discovery that can be 
                        used instead of static IP based discovery. For information on all 
                        options refer to our documentation: 
                        http://apacheignite.readme.io/docs/cluster-config
                    -->
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                        <property name="addresses">
                            <list>
                                <value>[[ hex.name ]]-n1.hex:47500</value>
                                <value>[[ hex.name ]]-n2.hex:47500</value>
                                <value>[[ hex.name ]]-n3.hex:47500</value>
                                <value>[[ hex.name ]]-n4.hex:47500</value>
                                <value>[[ hex.name ]]-n5.hex:47500</value>
                                <value>[[ hex.name ]]-n6.hex:47500</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>

Now, you have to ensure that your BigBoards app uses the new configuration on deployment. The Apache Ignite documentation on docker deployments provide the answer:

  • Go to your apache-ignite container definition on the BigBoards Hive
  • Set the environment variable CONFIG_URI to /etc/ignite/config/default-config.xml

The panel to define a container allow to pass environment variables

The outlined setup works because upon deployment:

  1. the git repository gets cloned to the target cluster
  2. the defined volume mapping gets mounted by using
    1. source the relative host directory ignite with respect to the config folder in the git repo; hence /config/ignite/ is the effective path as source folder
    2. target: the full directory /etc/ignite/config within the docker container as target folder
  3. the file default-config.xml becomes available in the container as /etc/ignite/config/default-config.xml
  4. the container picks up this file via the environment variable CONFIG_URI=/etc/ignite/config/default-config.xml

Redeploy your BigBoards app

To test your new configuration, you have to re-install your BigBoards app:

  1. Head over to your Hex's dashboard,
  2. uninstall any currently installed BigBoards app,
  3. go to the dashboard's library,
  4. install your new BigBoards app.
bb@optimus-prime-n1:~$ docker ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
66b711199185        index.docker.io/apacheignite/ignite:1.7.0   "/bin/sh -c $IGNITE_H"   2 minutes ago       Up 2 minutes                            apache-ignite

bb@optimus-prime-n1:~$ docker logs apache-ignite
[18:18:29]    __________  ________________
[18:18:29]   /  _/ ___/ |/ /  _/_  __/ __/
[18:18:29]  _/ // (7 7    // /  / / / _/
[18:18:29] /___/\___/_/|_/___/ /_/ /___/
[18:18:29]
[18:18:29] ver. 1.7.0#20160801-sha1:383273e3
[18:18:29] 2016 Copyright(C) Apache Software Foundation
[18:18:29]
[18:18:29] Ignite documentation: http://ignite.apache.org
[18:18:29]
[18:18:29] Quiet mode.
[18:18:29]   ^-- Logging to file '/opt/ignite/apache-ignite-fabric-1.7.0-bin/work/log/ignite-412babb5.0.log'
[18:18:29]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[18:18:29]
[18:18:29] OS: Linux 3.19.0-23-generic amd64
[18:18:29] VM information: OpenJDK Runtime Environment 1.8.0_102-8u102-b14.1-1~bpo8+1-b14 Oracle Corporation OpenJDK 64-Bit Server VM 25.102-b14
[18:18:29] Configured plugins:
[18:18:29]   ^-- None
[18:18:29]
[18:18:29] Security status [authentication=off, tls/ssl=off]
[18:18:30] Topology snapshot [ver=6, servers=6, clients=0, CPUs=24, heap=6.0GB]
[18:18:31] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[18:18:31]
[18:18:31] Ignite node started OK (id=412babb5)
[18:18:31] Topology snapshot [ver=6, servers=6, clients=0, CPUs=24, heap=6.0GB]

Conclusion

In this tutorial, you learned the basic components of a BigBoards app:

  1. General info contains the reference to the git configuration repository.
  2. Containers are the basic units of deployment.
  3. Groups are the deployment descriptor that define what containers get deployed where.
  4. Volume Mappings link the configuration repository to the Docker containers.

Assembling a set of technologies into a BigBoards app is super easy. I made a BigBoards app to deploy Apache Ignite while listening to Mathias Lavaert's presentation at the 39th BigData.be meetup!

However, as with all distributed technologies, things can get complicated fairly quickly! Your understanding of the technologies and the services in your BigBoards app are crucial for a clean and maintainable solution!

We are still learning. Every day!
Join us! Learn! Improve! Participate!

Share you feedback, so we can all grow together on technology. Data is hard enough on its own!!!

Wim Van Leuven
Wim Van Leuven
Big Data Enthusiast, Technology Leader, Hands-on I am passionate about software and solutions. Organizer of BigData.be, the Belgian community on big data and NoSQL. Co-founder of BigBoards.io.

Comments are closed.