Saturday, March 4, 2017

Sending logs from NodeMCU/Lua to Graylog2

Graylog2 is a great product. It might not be the most sophisticated one when it comes to log aggregation and interrogation, but it does its job very well. After employing Graylog2 to collect logs from all my Docker containers I thought that it would be absolutely fantastic to be able to manage logs from the IoT devices as well.

Installing the right input plugin

Before we can push stuff into Graylog2 we need to teach it the protocol of IoT devices - MQTT. Luckily enough there is a ready-made and supported input plugin just for that. To install it simply download it to the plugin folder in your Graylog2 installation and restart Graylog2. Done.

The next thing is to add an input for MQTT. Select System/Inputs and then create an input by selecting MQTT TCP (GEFL) from the drop-down list of available connectors. You need to specify the MQTT server and queue which the connector will subscribe to. In the example below I used system/logs so if you would go for a different name you need to adjust the code accordingly. And last but not least you need to start the input by pressing the Start input button. Done.

Programming in Lua

Lua and the NodeMCU environment is a beautiful thing. It is simple, powerful and it just rocks. For our little project we'll need to build ourselves a version with bmp085, cjson, wifi and mqtt modules. Then we need to download it to the module and we're all set to start coding in Lua.

The code here initializes WiFi connection, once the connection is established connects to the MQTT server, initializes the BMP085 sensor (here we're using pins D2-sda and D1-scl as per the documentation) and sets up a recurring alarm every second to read the sensor and send GEFL-formatted logs if MQTT has been connected. Easy.

Transferring files to NodeMCU

There are many great ways to send Lua files to the module. I like to use the one written in NodeJS called nodemcu-tool

$ npm install -g nodemcu-tool
$ nodemcu-tool upload init.lua
$ nodemcu-tool terminal

Now press the reset button and your NodeMCU should inform you of 3 things:

  1. That it connected to WiFi network and has an IP address
  2. That it has connected to the MQTT server
  3. That it is sending logs to MQTT server
There is not much more to it. Give it a try and if you stumble upon any challenges getting it to work feel free to leave a comment.

Happy logging!

Spring Boot application with Graylog2

I have been looking for an alternative to Splunk for a while and recently I have stumbled upon Graylog2. It is a very advanced tool well worth looking at. I decided to do a minimalist Spring Boot project to test its capabilities

The configuration

There are 2 steps required in order to get the logs to Graylog2:

  • Adding an artifact with GEFL appender
  • Adding the GEFL appender
I have tried 2 different GEFL appenders: Logback-gelf v0.10p1 which did work but didn't give me all the necessary details into Graylog2, and logback-gelf that worked for me very well.

I added the dependency to my pom.xml as follows

<dependency>
    <groupId>de.siegmar</groupId>
    <artifactId>logback-gelf</artifactId>
    <version>1.0.3</version>
</dependency>

And then added a logback configuration file to my resources

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/base.xml" />

    <appender name="GELF" class="de.siegmar.logbackgelf.GelfUdpAppender">
        <graylogHost>localhost</graylogHost>
        <graylogPort>12201</graylogPort>
        <layout class="de.siegmar.logbackgelf.GelfLayout">
            <originHost>localhost</originHost>
            <includeRawMessage>false</includeRawMessage>
            <includeMarker>true</includeMarker>
            <includeMdcData>true</includeMdcData>
            <includeCallerData>true</includeCallerData>
            <includeRootCauseData>true</includeRootCauseData>
            <includeLevelName>true</includeLevelName>
            <shortPatternLayout class="ch.qos.logback.classic.PatternLayout">
                <pattern>%m%nopex</pattern>
            </shortPatternLayout>
            <fullPatternLayout class="ch.qos.logback.classic.PatternLayout">
                <pattern>%m</pattern>
            </fullPatternLayout>
            <staticField>app_name:backend</staticField>
            <staticField>os_arch:${os.arch}</staticField>
            <staticField>os_name:${os.name}</staticField>
            <staticField>os_version:${os.version}</staticField>
        </layout>
    </appender>

    <root level="trace">
        <appender-ref ref="GELF" />  
        <appender-ref ref="CONSOLE" />
    </root>
</configuration>

This gave me all the logs into a local instance of Graylog2 running on Docker.

version: '2'
services:
  mongo:
    image: "mongo:3"
  elasticsearch:
    image: "elasticsearch:2"
    command: "elasticsearch -Des.cluster.name='graylog'"
  graylog:
    image: graylog2/server:2.2.2-1
    environment:
      GRAYLOG_PASSWORD_SECRET: somepasswordpepper
      GRAYLOG_ROOT_PASSWORD_SHA2: 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      GRAYLOG_WEB_ENDPOINT_URI: http://127.0.0.1:9000/api
    depends_on:
      - mongo
      - elasticsearch
    ports:
      - "9000:9000"
      - "12201/udp:12201/udp"
      - "1514/udp:1514/udp"

Happy logging!

Monday, January 23, 2017

Creating an LTSP Server on Linux Mint

I wanted to try this out for ages but somehow I have never done it. Last weekend I decided to give it a go and to create an LTSP server and a thin client that'll start over PXE with no disk, floppy, USB stick or CD/DVD. It turned out to be quite simple but a few quirks made me wonder for a bit what the hell is going on. Here's a log of those things.

DHCP starts, TFTP from a live CD works but TFTP from PXE times out

That happens when using NAT network in VirtualBox. They are broken. Use 2 cards on the server, one in whatever mode you want, that will connect to the internet (I use bridged or NAT - both work fine) and for the second card use an internal network.

The server needs to have them properly configured. Leave the first one that's bridged to get its config using DHCP and the second one configure to have a static IP address (e.g. 192.168.67.2), netmask of 255.255.255.0 and NOTHING ELSE. For the client setup just one network card connected to internal network (same as for the server) and leave everything else be.

By the way, I used 2 different types of virtual network cards: Intel for the bridged one and AMD for internal network. The AMD card starts faster over DHCP :)

The /etc/ltsp/dhcpd.conf file needs to be updated accordingly. For the 192.168.67.0 network I have configured mine as follows:

authoritative;

subnet 192.168.67.0 netmask 255.255.255.0 {
    range 192.168.67.20 192.168.67.250;
    option domain-name "example.org";
    option domain-name-servers 8.8.8.8;
    option broadcast-address 192.168.67.255;
    option subnet-mask 255.255.255.0;
    option root-path "/opt/ltsp/i386";
    if substring( option vendor-class-identifier, 0, 9 ) = "PXEClient" {
        filename "/ltsp/i386/pxelinux.0";
    } else {
        filename "/ltsp/i386/nbi.img";
    }
}
I have removed option routers ... and next-server [...] options because they are not necessary as I am not connecting the client to Internet (everything runs on the server) and I am running TFTP on the same host with DHCP server.

On Linux Mint "Reached target Graphical Interface." but nothing happens

This one is easy but not obvious if you have done LTSP on Ubuntu. After the initial creation of image (ltsp-build-client [...]) install virtualbox' X11 guest driver like so:

$ sudo ltsp-chroot -p -d apt-get install virtualbox-guest-x11
And then update the image as usual:
$ sudo ltsp-update-image
On Ubuntu-based servers those drivers get installed automatically. Very weird...

Fucked up client display

If you will see a display that is sort of hacked in half horizontally and then vertically with things looking like it's Picasso's fantasy then your client has either too much graphical memory, 3D acceleration enabled or too few MBs of RAM. I set my client VM to have 1GB RAM, 128MB video RAM and 3D acceleration enabled but your mileage might vary. Before you do anything just add RAM - that should help

The client started - what's the username and password?

This one is really tricky if you don't know what LTSP is all about. Essentially LTSP client is Linux with X11 and SSH client bound to your server. So whenever you are logging in use the same account as on server. If you need more users - add them to the server. And remember: all apps are not running on the client - they run on the server! This is why a Raspberry Pi works great as an LTSP client. I love it!

I have checked Linux Mint XFCE, Ubuntu 16.04.1, LUbuntu, Ubuntu Mate - they all work great. If you're a RedHat/Fedora fan you're on your own.

Happy LTSPing!

Saturday, November 12, 2016

Flashing NodeMCU

Howdy!

This is going to be a quick one. When flashing NodeMCU with new firmware you need to remember that esptool.py requires some params for it to work correctly. And let's not forget the flash needs to be erased before we do anything.

esptool.py --port /dev/ttyUSB0 erase_flash
esptool.py --port /dev/ttyUSB0 write_flash -fm dio -fs 32m -ff 40m \
  0x000000 nodemcu-integer.bin \
  0x3fc000 esp_init_data_default.bin

Let me know if it works for you! Oh and btw - you can build the firmware online - I'm loving it!

Happy flashing!

Friday, November 4, 2016

Vue js - finally a framework I feel comfortable using

The story so far...

I have switched to frontend development about 10 months ago. I met a lot new friends, interesting ideas and got back to a place where you don't really need to travel half way around the world to react on someone pressing a button. That was great up until I was acquainted with JavaScript framework that I was supposed to work with. Ember.JS. I mean the idea behind it is great, don't get me wrong, but it is the tiny little details that make it a nightmare to use. Features like mixins, mustache syntax with helpers, views in God knows which location, lots of decisions pre-made for the developer (like we're not smart enough to make them ourselves, doh!) and the sick Broccoli souse on top of it - I hate Ember with a passion!

And so naturally I started looking for an alternative. First at AngularJS, then Angular2 - none of those felt like a good choice at the time. At some point I found a presentation where a scared to death person on the stage was criticizing MVC on the front end saying that's actually the wrong thing to do. That was a presentation of ReactJS. Funny enough a long long time ago I have had a conversation with one of my colleges about putting MVC to use in Delphi applications. It ended quite rapidly with me saying You're insane! and I have a strong feeling that it took 8 years for the industry to catch up with my statement.

But ReactJS felt a bit odd. And I don't mean the JSX syntax - I actually love it! But the whole shebang with Redux, functional components, state-full components - it was just a lot to take in. On top of that you needed to acquaint yourself with Webpack which at least fueled my interest in React for quite some time because of the hot module reloading. I even gave a talk about Webpack on a conference - that's how strong I feel it's a good thing

Let there be light (at the end of a tunnel)

And than, one evening, out of the blue someone posted an article on LinkedIn about vue.js. I never heard that name before, I gave it a quick read, got interested but not nearly enough to continue reading that night. Still some itch remained after I read that it was basically a simple framework. I can do simple - I actually prefer that over complex stuff - that is why I hate both Ember and JSF. A couple of days later I came back to it, read the whole user's guide in one evening and basically couldn't believe how obvious it was. I don't mean just simple (vue.js concepts are quite sophisticated at times) but just obvious. It takes all that's best from all the previous frameworks and does the same but in the right way.

Since then I have started a major project in vue.js, scaffolded with vue-cli, developed in Atom, with Java/Spring backend and I basically loved everything about it. The shorthand syntax for assigning event handlers with the @ syntax is just so fricken better than having to have 2 methods one for hooking up those events and one for unhooking them. The shorthand syntax for binding fields to views - awesome! The fact that you can have one file per component (not HAVE-TO but CAN) is such a relief from navigating millions of files in different locations in Ember. Using just properties and not custom-made this.get and this.set methods makes the code just so much more readable. But still - it's not the biggest win from using vue.js. It's how the framework structures and enforces things that makes it completely out of this world by comparison to other frameworks

A short tale of data and props

One of the biggest fuckups in all other frameworks (and in Ember in particular) is the fact that you create the API of your component when you use it. This basically means that there is no single point where you actually can do it. Not that it should bother you when you create the next "Hello, world!" but when it comes to a big project with 30 developers in it, having no definition of an API for a components and no enforcement of defining it turns on productivity after short few weeks.

Short personal note: I have seen a presentation from a very nice girl from the Polimer project at Google where she stated that if you're going to design a new component's API make it work like a button - not like the input.

vue.js actually makes you define the API or you can't use it! It even warns you that props should actually be read-only. You get a warning in the console when you try to assign a value to it!. How cool and thoughtful is that?!

Carry on my wayward son!

I think vue.js is the first of great many frameworks I came across that actually captures what developers need to have to do their job in a professional and maintainable fashion. Back in the days there was ExtJS which has fallen into the MVC trap as well (don't know the current state of things with that one though) but otherwise none of the currently popular frameworks feel like they are good fit for the job. So if you can - try vue.js out! And if your experience is not as mine - share it in comments! I'd like to know why and if that will impact me as well.


Have fun!

Thursday, October 6, 2016

Speeding up NPM

NPM is slow. I mean it is really, really slow when downloading packages. Partially because it makes a ton of requests to remote servers. That can be made faster!

There are 2 options that you'll find useful:

They both create a local cache that npm is later on interact with instead of the remote repository.

npm_lazy

This one is really easy to setup and use. Just do
npm install -g npm_lazy
npm_lazy
and use npm with the local registry like so:
npm --registry http://localhost:8080 install

Nexus

This is a bit more complex. Nexus can be started as a stand-alone app quite easily but I would recommend using Docker as it is a far easier experience. So to start Nexus you type:
docker run -d -p 8081:8081 --name nexus sonatype/nexus
Then navigate to http://localhost:8081, login as admin/admin123 and create an npm-proxy. Select "Repositories" from the list on the left hand side, then from the "Add..." drop down at the top select "Add proxy repository", fill in as follows:
  • Repository ID - npm-registry-proxy
  • Repository Name = NPM Registry Proxy
  • Provider - npm
  • Remote Storage Location - https://registry.npmjs.org
and click on "Create repository".

Use it as follows:

npm --registry http://localhost:8081/content/repositories/npm-registry-proxy install

What's the gain?

Depending on the project you'll see a 30-50 percent speed boost when downloading packages. For the example project that I used here are the stats:
No proxy npm_lazy Nexus
1m43s 1m22s 57s

Have fun!

Edit on 23rd of January 2017: Forget all of the above. Just switch to Yarn and live happily ever after.

Wednesday, October 5, 2016

Guarding Ember's computed properties

Sometimes you hunt a bug that just won't show. It just literally won't allow itself to be found. Recently I've had one of those cases. This time it was just unimaginable what has actually happened. Someone, possibly by mistake, has overridden an Ember.ComputedProperty thus breaking the chain of updates. On top of that the app did updates to the model using websockets changing other properties of that model. This resulted in very strange flickering of values in an edit field. Very hard to diagnose.

After the bug was fixed I started looking at possible ways to guard it in the future and possibly to find other places where the same happens. There are 2 cases where this might happend:

  • using this.set('property', 'value');
  • using Handlebars template

One possible solution would be to reopen the Ember.Object class and to override the Ember.Object.set method. Unfortunately it only guards you from explicit calls to this.set(...) and all the overrides from Handlebars template remain silent.

But there is hope!

There is another class, Ember.ComputedProperty that is actually used as the tool to get and set computed properties. It also features a set method that can be monkey-patched to do our check:

(function() {
  var orgSet = Ember.ComputedProperty.prototype.set;

  Ember.ComputedProperty.prototype.set = function(obj, key) {
    if (obj[key] && obj[key]._getter && !obj[key]._setter) {
      console.error("Overriding computed property " +
        key + " on " + obj._debugContainerKey);
    }
    return orgSet.apply(this, arguments);
  };
})();

In this way if you override the computed property in a handlebars template or in the code you'll know about it. I only wish this would be part of the core framework. It would shave about 10 mandays of my project...

Here is an example project with the fix applied.

Have fun!

Edit on October 6th, 2016

I have been made aware of an Ember addon by Stefan Penner that addresses the same issue called ember-improved-cp. Although the idea of not using private API of the Ember.ComputedProperty is nice using that plugin is invasive and means using addon-provided macros instead of those that are provided straight from the framework which adds to the already high complexity of Ember-based projects. If you have a small enough project with few developers and you can afford that inconvenience you should probably go with the addon. If you need a plug-play-unplug solution you're going to get where you need to be with the monkey-patched version of Ember.ComputedProperty.prototype.set much faster.