Protocol Buffers
Protocol Buffers (aka “protobufs”) is another Google technology which integrates neatly with Java (and other languages). As their tag-lines says: “Protocol Buffers are language-neutral, platform-neutral extensible mechanisms for serializing structured data”.
For the sake of this example, we will look at 1) definition of a “proto” file; 2) integration with Bazel; 3) and use in a small Java test example.
Proto Message
At the heart of protobufs is the Message, which is an abstract language agnostic definition of a data class. As any class, the Message has fields, which can be of a few basic common native types or other Message types. Thus, any data structure can be realised.
For this example, let’s take the Person message from the official example. It comes with a few basic fields:
The proto file is compiled using the protoc
command line tool, which will produce language specific source code to depend on.
Bazel
For the sake of this example, we will use Bazel to build. It has native (more or less) support for protobufs. No extra dependencies are needed in the MODULE.bazel
nor the BUILD
file.
There are two rules to specify, one for the proto itself and one for the Java specific library. The latter will depend on the former:
This can now be compiled using:
Use in Java code
The generated class will be available for import as any other class. Notice the package, which is defined specifically for Java at the top of the .proto file.
A proto message can be instantiated using its builder methods. The instance is immutable.
The official example shows a slightly more complex structure, with nested objects:
Files from this example:
Bazel ❤ Java
Bazel is an open-source build and test tool similar to Make, CMake, Maven, and Gradle. It is cross platform, and as opposed to the mentioned tools, it supports a plethora of languages; many natively and almost anything else by extendable rules and tool chains.
Bazel’s strong points are a human-readable high-level build and dependency language (in contrast to Make or CMake), and having the dependency graph between modules and projects as the foundation of the build processing itself. This means that all actions, builds and tests can be cached deterministically, and only need to be re-executed if there are actual changes.
Finally, Bazel integrates seamlessly with external projects, source and project repositories, whether Git, Github, Pip, Maven repository, or simply downloaded tar.gz files. As is common, dependencies can be locked to specific versions and verified with check-sums.
Install
Installing Bazel is a breeze, and multiple alternatives are supported. My personal preference is to use their Debian / Ubuntu apt repository. When using Github Action, pre-built images and caching is readily available and easy to use, as seen here.
Build and test
To start using Bazel in an existing code base takes minimal setup. The bare minimum is 1) MODULE.bazel
file at top level repository directory (at first, it can be empty), and 2) at least one BUILD
file which defines a build rule, for example a unit test. These two commits put in place the top level files and the first BUILD file in our Git code.
A simple test rule might look like this:
It’s a Java Junit test, and it is given a name, which by convention is the same as the testcase file, and finally it specifies the source file(s).
Here’s another example, which defines a library, a binary, and a unit test. The two later depend on the library.
To build and execute these tests, various query strings can be used. The first builds and runs only a single test in the current directory:
To run all test at and below the current directory, the ...
query is used:
A //
at the beginning references the top level. Thus, this runs all tests:
To execute a binary, the run command is used:
For even more Bazel examples, including other languages, see our Bazel Examples repository.
Hello world with LibGDX
libGDX is a cross-platform Java game development library. It supports Windows, Gnu/Linux, Android, iOS, Mac, and web. The desktop variations use the Lightweight Java Game Library (LWJGL), with OpenGL support. There are already good comprehensive tutorials out there, so this article will only present the “Hello World” example with a small animation.
As opposed to basic OpenGL libraries, LibGDX and LWJGL offer a complete framework for creating a game, animations or interactions. To lower the bar to entry, the ApplicationListener interface has the common methods, most important the render() method, which is automatically called in a loop.
In the example below, a rotation matrix is defined and set, followed by drawing of the words “Hello World”. Because this method is called in a loop, the angle variable will update, and thus set a new rotation matrix every time. Similarly, the color is updated for each frame. The effect is a continuous never-ending animation.
There are multiple GDX packages, for different platforms and features, and for this example the following three are needed from the Maven Central repository. Be sure to include the natives-desktop version annotation on the gdx-platform package. For other platforms, e.g. Android, other packages are needed.
Here is the full code, as a stand-alone application.
TLS 1.2 over SUN HttpsServer
Security can be tricky, HTTPS and TLS no less so. There are many configuration details to be aware of, and the encryption algorithms and cipher suites are moving targets, with new vulnerabilities and fixes all the time. This example does not go into all these details, but instead shows a basic example of how to bring up a HTTPS server using a self-signed TLS 1.2 key and certificate.
The main component of Java TLS communication is the Java Secure Socket Extension (JSSE). A number of algorithms and cryptographic providers are supported. The central class is the SSLContext, supported by the KeyStore. These classes load and initialise the relevant keys, certificates, and protocols which are later used by a HTTPS server (or client). See the Oracle blog, for another brief introduction to TLS 1.2, and tips on diagnosing the communication. In particular, notice the unlimited strength implementations, which have to be downloaded separately, and copied to JAVA_HOME/lib/security.
The example below shows how the certificate and key are loaded from a Java KeyStore (.jks) file, and used to initialise the SSLContext with the TLS protocol. The SSLContext is passed to the SUN HttpsServer through a HttpsConfigurator. The HttpsServer implementation takes care of the rest, and the static file handler is the same as seen in the plain HTTP based example.
The code also includes a hard-coded generation of the key and certificate. A normal server would of course not implement this, but it’s included here to make the example self-contained and working out of the box. The following command is executed, and a Java KeyStore file containing a RSA based 2048 bits key, valid for one year. The keytool command will either prompt for name and organisational details, or these can be passed in using the dname argument. Also notice that password to the keystore and key are different. Further usage details on keytool can be found here.
Since the certificate is self-signed, a modern browser will yield a warning, and not allow the communication to continue without an explicit exception, as seen below. For the sake of this example, that is fine. If we do allow the certificate to be used, we will see that the communication is indeed encrypted, “using a strong protocol (TLS 1.2), a strong key exchange (ECDHE_RSA with P-256), and a strong cipher (AES_128_GCM)”, aka “TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”.
On the topic of keys and algorithms, Elliptic Curve Cryptography (ECC) is relevant. Digicert gives a brief introduction, with details on how to generate keys for Apache. Openssl has further information on EC using openssl.
The two files below is all that is needed. A key and certificate is generate if not already present under /tmp/test.jks. Go to https://localhost:9999/secure/test.txt and enable the security exception to try it out. Also notice the logging in the console window of the server.
Guava EventBus
Guava is Google’s utilities, collections and helpers library for Java. Similar in spirit to the Apache Commons set of libraries, but possibly a bit more pragmatic. Google’s engineers use the Guava components in their daily work, and have fine-tuned the APIs to boost their own productivity. The current release is version 21.
In this article, we’ll look at the Guava EventBus class, for publishing and subscribing to application wide events of all kinds. It strikes a good balance between convenience and type-safety, and is much more lightweight than rolling your own Event, Handler and Fire classes. See also the Guava documentations for further details.
To have an example to work with, the following Player skeleton class is implemented. It comes with three separate Event classes, which are very minimal in this contrived example. In a real application, they might inherit from a super-Event class, and possibly carry some content, at least the source of the event or similar. Alternatively, they could have been made simpler, by being elements of an enumeration, however, then we’d have to do manual matching and routing on the receiving end.
Notice, that the receiving part of the EventBus events is already in place, in the form of the @Subscribe annotations.
In the following code, the EventBus is created, and the Player is registered as a receiver. To publish events, all that is required is to pass objects of the same types to the EventBus.post() method. The internals of the EventBus will invoke all methods which are marked with the @Subscribe annotation and match the exact type. Event types are not inherited, so a catch-all subscriber is not possible, which is probably a good design choice.
The EventBus also makes unit testing easier, since sender and receiver can be decoupled without extensive mocking. In fact, we now have a choice of testing the methods of the Player class through the Eventbus, or by invoking its methods directly. Either might be fine when writing unit tests. Although, for component level test, sticking with the EventBus is probably better, as it will be closer to the live application.
To include the Guava library from the Maven Gradle repository, this will suffice.
Here is the full code listing.
Cantor circles and recursion
This article draws inspiration from an old post dubbed “Cantor Circles”, but which turned out to render bisecting circles rather than Cantor sets. Nevertheless, it gained some interest, and the exact image below, from the 2002 post, can now be easily found on Google Image search. In this post, the old code is revived, and animation and colors are added for nice patterns and fun. Cantor’s ternary set is also implemented.
I suspect the origin of the bisecting circles was a simple example for recursive code, and simply dividing by two makes the code and rendering easy to understand. In the block below, the function drawCircles() is recursive, calling itself twice with parameters to half the next circles. A recursive function needs a stopping condition, and in this case it is the parameter times which stops further recursions when it reaches 0. The helper function drawMidCircle() makes it more convenient to set the coordinates and size of the circle.
The next example adds animation. The recursion is the same, but adds an angle parameter to rotate the circles. Also notice that the angle a is negated, which leads to the alternating clockwise and counter-clockwise rotation within each small circle.
Also of interest here, is the Swing double buffering (actually, triple buffering is used in this example) using the BufferStrategy class. Notice that the paint() method of the Frame is no longer overridden to render the graphics, but rather the Graphics object is provided by the BufferStrategy and passed to the custom render() function.
The last class in this article adds colors, a few options and a spartan UI to tune them. The images and animation below give a quick impression of what is possible by adjusting the sliders. Also note, the color palette can be easily changed during runtime by pasting in hex based color string found on pages like colourlovers.com.
The GIF animations were created using ImageMagick, with commands like these:
Sun Doclet API
Since Java 1.2, the javadoc command has generated neatly formatted documentation. The tool comes with its own API which allows customised output. The relevant classes are under the com.sun package hierarchy, and located in JRE_HOME/lib/tools.jar, which typically will have to be included manually. E.g. it can be found under /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar.
Note that the Sun Doclet API is getting long in th tooth, and is already slated for replacement in Java 9, through the “Simplified Doclet API” in JDK Enhancement Proposal 221. Java 9 is planned for Q3 2017.
Meanwhile, the old Doclet API still does an OK job of parsing JavaDoc in .java source files. If the goal is to parse, rather than to produce the standard formatted JavaDoc, it’s useful to start the process pragmatically. Than can be achieved through its main class, com.sun.tools.javadoc.Main:
The execute() method will invoke the public static method start() in the specified class. In the example below, a few of the main JavaDoc entities are enumerated. The direct output can be see the block below. The class which is parsed is the example class itself, included at the bottom of this article.
- start main
Loading source file com/rememberjava/doc/SunDocletPrinter.java...
Constructing Javadoc information...
--- start
Class: com.rememberjava.doc.SunDocletPrinter
Annotation: @java.lang.Deprecated
Class tag:@author=Bob
Class tag:@since=123
Class tag:@custom=Custom Annotation
Class tag:@see="http://docs.oracle.com/javase/6/docs/technotes/guides/javadoc/doclet/overview.html"
Field: com.rememberjava.doc.SunDocletPrinter.SOME_FIELD
Method: com.rememberjava.doc.SunDocletPrinter.main(java.lang.String[])
Annotation: @java.lang.SuppressWarnings("Test")
Doc:
Raw doc:
@see "http://docs.oracle.com/javase/7/docs/technotes/guides/javadoc/standard-doclet.html"
Method: com.rememberjava.doc.SunDocletPrinter.start(com.sun.javadoc.RootDoc)
Doc: This method processes everything. And there's more to it.
Tag: Text:This method processes everything.
Param:root=the root element
Raw doc:
This method processes everything. And there's more to it.
@param root
the root element
@return returns true
--- the end
- done execute
Here is full file, which also shows the JavaDoc the example operates on.
Connecting to Bittorrent's Mainline DHT
In addition to its fast and efficient file transfer protocol, Bittorrent and other peer-to-peer file sharing networks bring another interesting technology to the masses: the Distributed Hash Table (DHT). In the case of Bittorrent, this is used to look up and download a torrent file based on its magnet link. The DHT network for Bittorrent is called Mainline DHT based the Kademlia DHT, although the network itself is separate from other implementations.
For Mainline DHT, there is an interesting open source client and library, called “mldht”. There are two instances on Github, with moreus/mldht as the original and the fork the8472/mldht. Although, both seems to be somewhat active. Both depend on the EdDSA library which the8472 has also forked. To get started, grab the source, and make sure the “mldht” code depend on or have access to the “ed25519” project. To confirm run the 28 unit tests from “mldht”. They should all pass.
The “mldht” project includes a stand-alone DHT server node which can be started through the executable Launcher class. Its main configuration is in “config.xml” which gets written to the current directory if it does not already exist. It’s based on “config-defaults.xml”. To be able to connect to the DHT network, I had to change the option “multihoming” to false in the config file.
Furthermore, in order to use the CLI client utility, the CLI server has to be enabled. I ended up with the following configuration file.
Once started, the activity can be observed in log/dht.log. There will be plenty of noise.
While the server is running, the CLI Client executable can be used to issue a few commands. Of interest is the “SAMPLE” command which lists random hash keys from random peers. Using that output, the “GETPEERS” can be used to look up specific hashes (make sure to remove the space formatting from the sample output). Finally, a torrent file can be downloaded with the “GETTORRENT” command.
Assuming the Java classpath is set correctly, example use would look like:
The hash used above is the Wikileaks “insurance” file posted last December, with the name “2016-12-09_WL-Insurance.aes256”. The “mldht” project does not contain any tools to actually read the torrent, but we can use the Transmission client:
The expected output would look like this:
Given that this worked fine, I’m thinking it should be trivial to create a custom ML DHT client which performs the steps above. I hope to come back to that in a future article.