Connecting to Bittorrent's Mainline DHT
In addition to its fast and efficient file transfer protocol, Bittorrent and other peer-to-peer file sharing networks bring another interesting technology to the masses: the Distributed Hash Table (DHT). In the case of Bittorrent, this is used to look up and download a torrent file based on its magnet link. The DHT network for Bittorrent is called Mainline DHT based the Kademlia DHT, although the network itself is separate from other implementations.
For Mainline DHT, there is an interesting open source client and library, called “mldht”. There are two instances on Github, with moreus/mldht as the original and the fork the8472/mldht. Although, both seems to be somewhat active. Both depend on the EdDSA library which the8472 has also forked. To get started, grab the source, and make sure the “mldht” code depend on or have access to the “ed25519” project. To confirm run the 28 unit tests from “mldht”. They should all pass.
The “mldht” project includes a stand-alone DHT server node which can be started through the executable Launcher class. Its main configuration is in “config.xml” which gets written to the current directory if it does not already exist. It’s based on “config-defaults.xml”. To be able to connect to the DHT network, I had to change the option “multihoming” to false in the config file.
Furthermore, in order to use the CLI client utility, the CLI server has to be enabled. I ended up with the following configuration file.
Once started, the activity can be observed in log/dht.log. There will be plenty of noise.
While the server is running, the CLI Client executable can be used to issue a few commands. Of interest is the “SAMPLE” command which lists random hash keys from random peers. Using that output, the “GETPEERS” can be used to look up specific hashes (make sure to remove the space formatting from the sample output). Finally, a torrent file can be downloaded with the “GETTORRENT” command.
Assuming the Java classpath is set correctly, example use would look like:
The hash used above is the Wikileaks “insurance” file posted last December, with the name “2016-12-09_WL-Insurance.aes256”. The “mldht” project does not contain any tools to actually read the torrent, but we can use the Transmission client:
The expected output would look like this:
Given that this worked fine, I’m thinking it should be trivial to create a custom ML DHT client which performs the steps above. I hope to come back to that in a future article.
Try Catch Exception
This post looks at the syntax variations of the try/catch/finally blocks. For further details, see the excellent Java tutorial on the topic.
To start off, below is the basic syntax, with code surrounded by a try-block, and a NullPointerException caught by the catch-block. As can bee seen, the code will fail, since the variable “str” is null, leading to a NullPointerException. The Exception variable in the catch-block is by most common conventions simply “e”. It has a few convenience methods, including printStackTrace() which shows the call trace since the Exception was thrown. Although the print-out might look scary, it does provide useful information to the developer. Thus, keeping the full stack trace is helpful, typically in a detailed log-file. That is beyond the scope of this post.
Exceptions are typed classes, and in the following example the catch-block will not be reached since the expected Exception is not the same or a sub-type of the one which is thrown: NullPointerException vs. ArithmeticException. Instead, the ArithmeticException will be thrown out of the method.
To handle multiple exception types, there are three options: Declare multiple catch-blocks with different types, as seen in the first part below; or declare multiple Exceptions within the same catch statement, as in the second part. The latter syntax has been available since Java 7. Finally, it’s possible to catch multiple Exceptions by specifying a type higher up the class hierarchy, e.g. using Exception or Throwable.
In addition to the try and catch blocks, there is also a finally-block. It is executed at the end, regardless of whether there was an Exceptions thrown or not. This is useful for setting state or cleaning up, and a common example is closing an IO stream. However, as seen below, this can get crufty since we have to consider that the IO object might not have been opened in the first place, and that the close() method throws its own checked Exception.
To clean up the code above, the try-with-resources syntax was introduced in Java 7. It allows the try statement to take an extra block which is executed before its content. In addition, variables declared within this block will be closed at the end through the Closeable interface. This significantly reduces the foot-print of the code above.
A good example for multiple resources is the Socket example discussed previously. Here the Socket and both IO streams are closable resources handled by the try-block.
Finally, a word on messaging and wrapping of Exceptions. As mentioned in the tutorial, it’s poor practice to throw RuntimeExceptions or simply wrap checked Exceptions, as seen below. However, regardless of where you stand in that debate, Exceptions can always be made more helpful and useful by clear messaging and relevant context. The wrapped RuntimeException below adds a more specific message and also includes the filename the IO stream operates on, since it might not be included in all types of IOExceptions. Furthermore, in the case of the File object, is is useful to use the getAbsolutePath() method. It forces the full path to the resolved and included. It really helps when debugging issues where the full path can be copy/pasted and confirmed.
Here is the full listing with all examples as tests.
Unit test Exceptions
Testing the “happy path” of the code, when everything goes right, is fine, however error handling and Exceptions is just as much a part of the code under test. In fact, it is possibly a more delicate area, since you want an application which degrades gracefully in the event of error. This post goes through different ways of setting expectations on thrown Exceptions.
In the first method below, the old style pre-Junit 4 way of asserting for an Exception is shown. The idea is that the Exception must be thrown, so if the execution reaches the fail() statement, that did not happen. The expected path is instead that the catch-block engages, with the expected Exception type. If a different type of Exception, which is not a sub-class of the caught Exception is thrown, it propagates out of the method and the test fails. Although this style is a bit clunky and verbose, it has a few advantages over the annotation-style: You have control over exactly what point in the code you expect the Exception to be thrown; you can inspect the Exception and assert its message; you can set a custom error message.
Since Java 6 and Junit 4, annotations become available, and the @Test annotation is now the way to declare a test method. It comes with an extra parameter expected which takes a Class type indicating which Exception is expected to be thrown. This approach is clean and even minimalist. If all there is to a test is a one-liner like shown below, this is a perfectly fine way to declaring the expectation. However, it loses the ability to inspect the message or cause. Furthermore, if the test method contains more lines, there is no way to control or verify from where the Exception originated.
Enter JUnit 4.7, and the @Rule annotation and ExpectedException rule. It solves the problem with the Test annotation above, but retains a clean way of expressing the assertion. In the example below, the Rule annotation makes sure the thrown field is initialised a-new before every test method. It can then be used right above the method which is expected to throw an Exception, and can assert on its type and message. The expectMessage() method asserts that the message contains rather than equals the expected string.
The ExpectedException class comes with some extra assertion methods which takes the popular Hamcrest matchers. In the code below, the endsWith matcher is used.
Finally, the expectCause() method takes another Hamcrest matcher, where for example the type of the contained cause can be asserted. Notice that the outer exception type can be asserted as well. The expectCause assertion only goes one level deep, so if further unnesting is required, a custom Matcher could be implemented. In this example, the wrapping RuntimeException does not alter the message, so it can be asserted directly. If the outer Exception has its own message, another custom Matcher would be needed to assert on the inner message.
Here is the full test case listing.
Socket client / server example
After posts on several server libraries, including Sun’s HTTP; Simple Framework’s HTTP; and Websocket, it seems appropriate to include a plain TCP socket server / client example. Without any application protocol to adhere to, it’s straight forward to setup and test. There are two essential classes involved: On the server side there’s a ServerSocket, while the client has a Socket. The only difference is that the ServerSocket accepts one or more incoming connections, and provide a Socket handle for each. After that, both client and server are the same, in that the Socket object provides input and output streams once established.
In the code below, a ServerSocket is set to listen to a specific port, and to accept a single incoming connection. It grabs the IO streams, makes these available to the rest of the test, and finally releases a semaphore lock to allow the test to continue. Normally, the server thread would wait for further incoming requests, and would probably spawn or assign pooled threads to handle the request. Here we focus only on the basic IO parts.
On the client side, it’s just as simple: Establish a connection to the severer host and port, and get the IO streams.
The rest of the test code below asserts that messages are received correctly both on the client and the server, using the raw streams and wrapped PrinterWriter helpers. Again, he semaphore is used to wait for the server thread to establish its connection before the rest of the test continues. Without it, using the IO streams will results in NullPointerExceptions, since they are not initialized yet.
Styles with JTextPane
The Swing tutorial for the JTextPane and demo code is good and comprehensive, covering multiple ways to interact with the the StyledDocument and add Styles. So, without repeating all of that, here’s a minimal example including only simple style changes on the insert update and selection events.
In the first snippet below, the implementation for the insertUpdate event is shown. For each typed character or pasted text, the entire text of the document will be searched for the word “foo”. For each occurrence, the bold attribute is set for that word. Notice that the update of the attribute happens on a separate AWT thread.
The other functionality of this small application is on select. If the selected word is “bar”, it is set to italic. Notice the dot and mark positions, which might be at the start or the end of the selection. Furthermore, notice that here the predefined ItalicAction from the StyledEditorKit is used, since we’re dealing with a selection. We just have to translate the CaretEvent into an ActionEvent, or rather just make sure to forward the source component of the selection. (The alternative would have been to go with the plain update as in the example above).
Here’s the full file listing, as a stand-alone application.
Line numbers for JEditorPane and JTextPane
There are many ways to add line numbers to the JEditorPane and JTextPane Swing components. This post looks at two solutions: One providing a custom ParagraphView which paints a child element that contains the line number for that paragraph. The other implements a separate component which is passed as the RowHeaderView of the JScrollPane. Both support scrolling and line wrapping, and in this example the latter includes current line highlighting, as seen in the image below.
ParagraphView
The ParagraphView solution is inspired by Stanislav Lapitsky’s old post on the topic, although with some improvements. The basic idea is to paint the line number in the child of the ParagraphView, using the paintChild() method. The advantage is that the relationship between the line and the line number is already established, so alignment is easy. We just have to count which element we’re dealing with to know which number to print. The two methods below show this part. Notice also that only the index 0 of the ParagraphView should be used for the line number, so we don’t count a wrapped line twice. Furthermore, a mono-spaced font helps with padding and alignment.
In order to create this custom ParagraphView, it has to be returned through a ViewFactory, which again has to be provided through an EditorKit. The file below shows the full implementation. Notice how the default ViewFactory is used for all other elements; only the Paragraph Element gets a custom view.
setRowHeaderView
The second solution renders the line numbers in a completely separate component, which can be a JComponent (or even an old style AWT Component). This component is passed in to the setRowHeaderView() method of JScrollPane. That way, the coupling becomes a bit cleaner than overriding multiple classes and methods as with the the ParagraphView solution. However, the down-side is that we must do the alignment with the Editor component manually. Luckily, the Swing Text Component API provides a solid API for this purpose.
The code below is inspired by Rob Camick’s article, but simplifies several aspects of his stand-alone API style class. At the centre is the translation between the Document model and its onscreen view. The viewToModel() method translates an x,y point on the screen to the nearest character offset, while the modelToView() method goes the other way. That way we can align the line number at the same y point as the line in the editor. In Camick’s code, he loops through the text by character offsets rather than paragraphs; the getRowEnd() method helps by finding the end of a line. Camick also points out the the line number should take the font decent of the editor font into account when positioning the line number. The FontMetrics class helps with measuring sizes of fonts.
Finally worth noting is the synchronisation between the editor and the margin component. Since they are separate, this must be handled through event listeners registered with the editor component. The DocumentListener and ComponentListener notifies of edit, movement and resizing events, while the CaretListener signals movement in the cursor position. All of these events force a repaint, but through a scheduled AWT thread, to make sure the editor view has updated first, as seen in the documentChanged() method at the bottom of the code section below.
Here are the full code listings for both examples, as stand-alone applications.
Simple Framework Websocket server
Following the previous post about a HTTP server based on the Simple Framework library, here’s the Websocket use case. The example gets a bit more complicated with both a HTTP handler, Websocket handler and client, but still a lot simpler than the original example code.
Two interfaces have to be implemented to have a meaningful Websocket server: First, the Service interface, which gets invoked when a new connection and Session is established. From here it is possible to register a FrameListener on the session channel, which is the second interface to implement. The FrameListener methods will get invoked when new Frames, whether it is data or control codes is received. This listener also gets notified about errors and when the session or socket is closed. Two minimal examples of these implementations are shown below, followed by the setup() method which ties it all together and starts the server. Notice how RouterContainer takes both the HTTP and Websocket handlers, and routes the incoming requests appropriately.
To test the core functionality of the Websocket server, a bit more is needed on the client side. First, a HTTP like handshake has to be requested and the response read, then the client must respond to ping control frames, and finally, it can send frames of data itself.
Below is the method which sends the handshake to the server as specified in the Websocket RFC 6455. There’s a few things to note: Currently, details like target path, the host, origin are hard-coded. In a normal client, they should be resolved and set in a proper way. The security key is also hard-coded, but should be generated according to the specification. Finally, note that the line ending has to be “\r\n” or 0xd, 0xa in hex. Therefore, when using a PrintWriter to wrap the raw OutputStream, be careful not to use the println() methods as they might append the incorrect line termination.
What’s useful about a minimal test client like this, is that’s it becomes easy to try out these kind of details in isolation, without distractions from other functionality. Although this example does not go as far as asserting the behaviour, it could easily be extended to an in-memory integration test of the server.
Finally, this test method ties the client together: it first opens the socket connection to the server; gets the IO streams; sends and receives the handshake; starts the ping response handler on a separate thread; sends one frame of its own with a ping, waits for some seconds to observe the server in operation, and finally sends a close operation code before closing the connection.
Here’s the full test case listing, which also includes a HTTP request test. A dependency on the Simple Framework library is needed, for example through the Gradle dependency:
Simple Framework HTTP server
A while back, I ranted about the Simple Framework example code and its unnecessary complexity. It turns out that their HTTP server library is indeed simple, if you just ignore the Spring setup and their example. In fact, as this static file server example shows, it’s very similar to the “hidden” Sun HTTP handler API.
The only interface which has to be implemented is the Container, which is equivalent to the request HttpHandler in the Sun HTTP library. The handle() method takes a normal HTTP request, and the response code and data can be written via the Response object. The first code block below shows a simple handler.
The next block shows how to tie the required Simple Framework classes together to start the server. The Container just mentioned goes into a ContainerSocketProcessor where concurrent processing can be setup up. The processor goes into a SocketConnection which connects to a socket on a given host and port, and the server runs.
Here is the full server example, started and verified as a unit test.