Send and Receive money with bitcoinj

15 January 2017

The bitcoinj library is easy to use for Bitcoin wallet and transaction functions for both native Java and Android applications. Although there are certain features missing, it seems mature enough to be included in a Bitcoin walled app or service.

Sometimes the source code leaves a bit to be desired in structure and readability: anonymous inner classes and other deep nesting blocks sometimes makes it difficult to follow; inheritance is often used where composition would have been be better; the Collections classes could have been used over arrays in many places. All of this might come back to haunt the developers later, for now they seem to be plowing on.

At least the basics are straight forward. The following code will read a test walled from disk, or create a new one if it does not already exist. The TestNet3 block chain and network is used. Since the bitcoinj library relies heavily on the Google Guava (com.google.common) classes, there are frequent artifacts of the threading and callback handling showing up. In this example, we want the code to block and wait, therefore the extra await-functions are required.

  public void testSync() {
    params = TestNet3Params.get();
    kit = new WalletAppKit(params, new File("/dev/shm"), "test");

    kit.startAsync();
    kit.awaitRunning();

    kit.stopAsync();
    kit.awaitTerminated();
  }

The nice thing about the test block chain is that it is a real public live block chain, with miners and a block chain explorer, but with no value in the coins. In fact, you can get free coins to test with from faucet.xeno-genesis.com or tpfaucet.appspot.com. (The latter has been timing out over the last days).

To get some free test coins, run the following code, wait for the prompt which shows the next receiving address, and head over to faucet.xeno-genesis.com to ask them to send some money there. It should show up as received within a few seconds. Your wallet now contains some coins.

  public void disabled_testReceive() {
    startSync();

    wallet.addCoinsReceivedEventListener(this::coinsReceived);

    Address toAdr = wallet.currentReceiveKey().toAddress(params);
    System.out.println("Waiting to receive coins on: " + toAdr);

    while (!receivedCoins) {
      sleep(100);
    }
  }

  private void coinsReceived(Wallet wallet, Transaction tx, Coin prevBalance, Coin newBalance) {
    Coin value = tx.getValueSentToMe(wallet);
    System.out.println("Received tx for " + value.toFriendlyString() + ": " + tx);
    receivedCoins = true;
  }

Since the test network is a real network with real miners, it’s good etiquette to return your test coins to the pool for others to use once you’re done with them. The following code takes care of that, returning them to the TP Faucet default return address “n2eMqTT929pb1RDNuqEnxdaLau1rxy3efi”. You can return all your coins, or just a fraction if you want to experiment more. This will also wait a few seconds for the callback confirmation.

  public void disabled_testSend() throws InsufficientMoneyException {
    startSync();

    // Adjust how many coins to send. E.g. the minimum; or everything.
    Coin sendValue = Transaction.REFERENCE_DEFAULT_MIN_TX_FEE;
    // Coin sendValue = wallet.getBalance().minus(Transaction.DEFAULT_TX_FEE);
    
    Address sendToAdr = Address.fromBase58(params, TPFAUCET_RETURN_ADR);
    SendRequest request = SendRequest.to(sendToAdr, sendValue);

    SendResult result = wallet.sendCoins(request);

    result.broadcastComplete.addListener(() -> {
      println("Coins were sent. Transaction hash: " + result.tx.getHashAsString());
      sentCoins = true;
    }, MoreExecutors.sameThreadExecutor());

    while (!sentCoins) {
      sleep(100);
    }
  }

Finally, it’s worth noting that bitcoinj is a “live” library, in development and with the latest update available through Gradle. To make this work, there’s a few settings and dependencies to take care of. The logging framework used by bitcoinj is SL4J, and an actual implementation library (e.g. “sl4j-simple”) is also need. It can be downloaded, or included as a Gradle build dependency as seen below.

Then, your source code directory structure might not match the default Gradle “main”, “test” structure. My current structure keeps all source code under the directory “src”, so I have specified that.

Gradle integrates OK with Eclipse, but be careful with the “refresh” option. It tends to insist on changing the classpath setting of the project, so the packages disappear. It’s a good idea to keep the .classpath setting file under version control.

apply plugin: 'java'

sourceSets {
  main {
    java {
      srcDirs = ['src']
    }
  }
}

repositories {
  jcenter()
}

dependencies {
  compile 'org.slf4j:slf4j-api:1.7.21'
  compile 'org.slf4j:slf4j-simple:1.7.21'
  compile 'org.bitcoinj:bitcoinj-core:0.14.3'

  testCompile 'junit:junit:4.12'
}

The following listing shows all the tests. It demonstrates similar functionality as seen in the ForwardingService class in the main bitcoinj getting started guide. Hopefully, the code is a bit easier to read and run this way.

BitcoinjApiTest.java
GitHub Raw
/* Copyright rememberjava.com. Licensed under GPL 3. See http://rememberjava.com/license */
package com.rememberjava.bitcoin;

import java.io.File;

import org.bitcoinj.core.Address;
import org.bitcoinj.core.Coin;
import org.bitcoinj.core.InsufficientMoneyException;
import org.bitcoinj.core.NetworkParameters;
import org.bitcoinj.core.Transaction;
import org.bitcoinj.kits.WalletAppKit;
import org.bitcoinj.params.TestNet3Params;
import org.bitcoinj.wallet.SendRequest;
import org.bitcoinj.wallet.Wallet;
import org.bitcoinj.wallet.Wallet.SendResult;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.slf4j.impl.StaticLoggerBinder;

import com.google.common.util.concurrent.MoreExecutors;

/**
 * Multiple large file and network I/O tests for the Testnet3 block chain.
 * Comment in and out @Test annotations as need.
 * 
 * For more info:<br>
 * https://en.bitcoin.it/wiki/Testnet<br>
 * 
 * For test coins: <br>
 * http://faucet.xeno-genesis.com <br>
 * http://tpfaucet.appspot.com <br>
 * 
 * For test block explorer: <br>
 * https://testnet.blockexplorer.com
 *
 */
public class BitcoinjApiTest {

  /**
   * http://tpfaucet.appspot.com
   */
  private static final String TPFAUCET_RETURN_ADR = "n2eMqTT929pb1RDNuqEnxdaLau1rxy3efi";

  private static final File WALLET_DIR = new File("/tmp");

  private static final String WALLET_PREFIX = "rjtest";

  private NetworkParameters params;

  private WalletAppKit kit;

  private Wallet wallet;

  private volatile boolean receivedCoins;

  private volatile boolean sentCoins;

  /**
   * For all tests, connect to the TestNet3 network.
   */
  @Before
  public void setup() {
    params = TestNet3Params.get();
    kit = new WalletAppKit(params, WALLET_DIR, WALLET_PREFIX);
  }

  /**
   * Connect, sync and wait till done.
   */
  private void startSync() {
    kit.startAsync();
    kit.awaitRunning();
    sleep(1000);

    wallet = kit.wallet();
  }

  /**
   * Disconnect and wait till done.
   */
  @After
  public void teardown() {
    kit.stopAsync();
    kit.awaitTerminated();
  }

  /**
   * Even though it is possible to run without the logger implementation classes
   * in place, it is very confusing and misleading.
   * 
   * The bitcoinj logs output connection and transaction information. Without
   * them, it's difficult to know what is happening, and the application might
   * seem "stuck".
   */
  @Test
  public void checkLogger() {
    StaticLoggerBinder.getSingleton();
  }

  /**
   * Connects to the TestNet3 network and disconnects. If there is no walled
   * stored at the specified location, a new one is created and relevant blocks
   * are downloaded. This can take 1 to 2 minutes. If information is stale, it
   * can also take a minute to update.
   * 
   * The wallet used in this test is separate from the other tests.
   */
  @Test
  public void testSync() {
    params = TestNet3Params.get();
    kit = new WalletAppKit(params, new File("/dev/shm"), "test");

    kit.startAsync();
    kit.awaitRunning();

    kit.stopAsync();
    kit.awaitTerminated();
  }

  /**
   * Connects and prints address information about the test wallet (from the
   * TestNet3 network).
   */
  @Test
  public void printAddresses() {
    startSync();

    System.out.println("Receive Addresses:");
    wallet.getIssuedReceiveAddresses().stream().forEach(this::println);

    System.out.println("Watched Addresses:");
    wallet.getWatchedAddresses().stream().forEach(this::println);

    System.out.println("Current change address:");
    System.out.println(wallet.currentChangeAddress());
  }

  /**
   * Connects and prints wallet information (from the TestNet3 network).
   */
  @Test
  public void printWalletInfo() {
    startSync();

    Coin balance = wallet.getBalance();
    println("Balance satoshis: " + balance.toString());
    println("Balance friendly: " + balance.toFriendlyString());
    println("Version: " + wallet.getVersion());
  }

  /**
   * Waits for coins to be received on the prompted address. This test blocks
   * until the coinsReceived callback is called.
   * 
   * (This test is commented out to avoid blocking the other tests).
   */
  // @Test
  public void disabled_testReceive() {
    startSync();

    wallet.addCoinsReceivedEventListener(this::coinsReceived);

    Address toAdr = wallet.currentReceiveKey().toAddress(params);
    System.out.println("Waiting to receive coins on: " + toAdr);

    while (!receivedCoins) {
      sleep(100);
    }
  }

  /**
   * Callback called when the coins have been received.
   */
  private void coinsReceived(Wallet wallet, Transaction tx, Coin prevBalance, Coin newBalance) {
    Coin value = tx.getValueSentToMe(wallet);
    System.out.println("Received tx for " + value.toFriendlyString() + ": " + tx);
    receivedCoins = true;
  }

  /**
   * Sends test coins back to the TPFAUCET test network address. This test
   * blocks until the broadcastComplete callback is called.
   * 
   * (This test is commented out to avoid blocking the other test and
   * inadvertently sending away our money.
   */
  // @Test
  public void disabled_testSend() throws InsufficientMoneyException {
    startSync();

    // Adjust how many coins to send. E.g. the minimum; or everything.
    Coin sendValue = Transaction.REFERENCE_DEFAULT_MIN_TX_FEE;
    // Coin sendValue = wallet.getBalance().minus(Transaction.DEFAULT_TX_FEE);
    
    Address sendToAdr = Address.fromBase58(params, TPFAUCET_RETURN_ADR);
    SendRequest request = SendRequest.to(sendToAdr, sendValue);

    SendResult result = wallet.sendCoins(request);

    result.broadcastComplete.addListener(() -> {
      println("Coins were sent. Transaction hash: " + result.tx.getHashAsString());
      sentCoins = true;
    }, MoreExecutors.sameThreadExecutor());

    while (!sentCoins) {
      sleep(100);
    }
  }

  private void sleep(long millis) {
    try {
      Thread.sleep(millis);
    } catch (InterruptedException e) {
      e.printStackTrace();
    }
  }

  private void println(Object str) {
    System.out.println(str);
  }
}

MIDI basics

13 January 2017

Working with MIDI in Java is easy. The standard API have classes covering MIDI file I/O, device I/O, sequencing and sound synthesis. This comprhensive tutorial covers most aspects. It also helps to know something about the MIDI format. Juan Bello’s slide deck is an excellent introduction. midi.org is also a good source, including their full message reference table.

To play a single C note through the default included “Gervill” soft synthezier, the following snippet will do. As commented in the code, there’s come discrepancy on pitch or octave labelling, but the 60th note is still on the save octave as A at 440 Hz.

  public void testPlayNote() throws InvalidMidiDataException, MidiUnavailableException {
    // 60 = middle C, C4 (or C3 or C5)
    // https://en.wikipedia.org/wiki/Scientific_pitch_notation#Similar_systems
    int channel = 0;
    int note = 60;
    int velocity = 127; // velocity (i.e. volume); 127 = high
    ShortMessage msg = new ShortMessage();
    msg.setMessage(ShortMessage.NOTE_ON, channel, note, velocity);

    long timeStamp = -1;
    MidiDevice synthesizer = getSynthesizer("Gervill");
    synthesizer.open();
    Receiver receiver = synthesizer.getReceiver();
    receiver.send(msg, timeStamp);
    receiver.close();
  }

This gives a brief overview of the MIDI devices on the system, both software based and hardware devices. Depending on drivers and OS, the various devices might show up under different names and types.

  public void testGetMidiDeviceInfo() throws MidiUnavailableException {
    Info[] infos = MidiSystem.getMidiDeviceInfo();
    for (Info info : infos) {
      MidiDevice device = MidiSystem.getMidiDevice(info);
      boolean isSequencer = device instanceof Sequencer;
      boolean isSynthesizer = device instanceof Synthesizer;

      System.out.println(
          "Info: " + device + ", '" + 
              info.getName() + "',  '" + 
              info.getVendor() + "', '" + 
              info.getVersion() + "', '" + 
              info.getDescription() + "', " + 
              "sequencer=" + isSequencer + 
              ", synthesizer=" + isSynthesizer);
    }
  }

Finally, the following methods demonstream MIDI file I/O.

  public void testWriteFile() throws InvalidMidiDataException, IOException {
    int volume = 0x70;
    int tick = 0;

    Sequence sequence = new Sequence(Sequence.PPQ, 24);
    Track track = sequence.createTrack();

    MetaMessage meta = new MetaMessage();
    String trackName = "First track";
    meta.setMessage(0x03, trackName.getBytes(), trackName.length());
    track.add(new MidiEvent(meta, tick));

    int notes[] = IntStream.range(60, 73).toArray();

    for (int note : notes) {
      ShortMessage on = new ShortMessage(ShortMessage.NOTE_ON, note, volume);
      track.add(new MidiEvent(on, tick));
      tick += 4;

      ShortMessage off = new ShortMessage(ShortMessage.NOTE_OFF, note, 0);
      track.add(new MidiEvent(off, tick));
      tick += 4;
    }

    MidiSystem.write(sequence, 1, MIDI_FILE);
  }

  public void testReadFile() throws InvalidMidiDataException, IOException {
    Sequence sequence = MidiSystem.getSequence(MIDI_FILE);
    
    System.out.println("tracks: " + sequence.getTracks().length);
    System.out.println("patches: " + sequence.getPatchList().length);
    
    for (Track track : sequence.getTracks()) {
      int size = track.size();
      System.out.println("track size: "+size);
      
      IntStream.range(0, size).boxed()
        .map(track::get)
        .map(RawMessage::new)
        .map(AbstractMessage::getNoteOctave)
        .map(str -> str + " ")
        .forEach(System.out::print);
      System.out.println();
    }
  }

  public void testMidiFileFormat() throws InvalidMidiDataException, IOException {
    MidiFileFormat format = MidiSystem.getMidiFileFormat(MIDI_FILE);

    System.out.println("bytes: " + format.getByteLength());
    System.out.println("length ms: " + format.getMicrosecondLength());
    System.out.println("divisionType: " + format.getDivisionType());
    System.out.println("resolution: " + format.getResolution());
    System.out.println("type: " + format.getType());
    System.out.println("properties: " + format.properties().size());

    format.properties().entrySet().stream()
      .map(e -> e.getValue() + "=" + e.getKey())
      .forEach(System.out::println);
  }

The full test case can be seen below. There’s a few more helper classes and details in the same package here. Then there’s some special implementation and details for the Roland TB-03 and TR-8 devices here.

MidiApiTest.java
GitHub Raw
/* Copyright rememberjava.com. Licensed under GPL 3. See http://rememberjava.com/license */
package com.rememberjava.midi;

import static com.rememberjava.midi.MidiUtils.filteredDeviceStream;
import static com.rememberjava.midi.MidiUtils.onClassnameEquals;

import java.io.File;
import java.io.IOException;
import java.util.stream.IntStream;

import javax.sound.midi.InvalidMidiDataException;
import javax.sound.midi.MetaMessage;
import javax.sound.midi.MidiDevice;
import javax.sound.midi.MidiDevice.Info;
import javax.sound.midi.MidiEvent;
import javax.sound.midi.MidiFileFormat;
import javax.sound.midi.MidiSystem;
import javax.sound.midi.MidiUnavailableException;
import javax.sound.midi.Receiver;
import javax.sound.midi.Sequence;
import javax.sound.midi.Sequencer;
import javax.sound.midi.ShortMessage;
import javax.sound.midi.Synthesizer;
import javax.sound.midi.Track;
import javax.sound.midi.Transmitter;

import org.junit.Test;


/**
 * See also: https://docs.oracle.com/javase/tutorial/sound/overview-MIDI.html
 */
public class MidiApiTest {

  private static final String SOFT_SYNTHESIZER = "SoftSynthesizer";

  private static final String MIDI_FILE_PATH = "com/rememberjava/midi/test.mid";

  private static final File MIDI_FILE = new File(MIDI_FILE_PATH);

  @Test
  public void testGetMidiDeviceInfo() throws MidiUnavailableException {
    Info[] infos = MidiSystem.getMidiDeviceInfo();
    for (Info info : infos) {
      MidiDevice device = MidiSystem.getMidiDevice(info);
      boolean isSequencer = device instanceof Sequencer;
      boolean isSynthesizer = device instanceof Synthesizer;

      System.out.println(
          "Info: " + device + ", '" + 
              info.getName() + "',  '" + 
              info.getVendor() + "', '" + 
              info.getVersion() + "', '" + 
              info.getDescription() + "', " + 
              "sequencer=" + isSequencer + 
              ", synthesizer=" + isSynthesizer);
    }
  }

  @Test
  public void testGetSequencer() throws MidiUnavailableException {
    Sequencer sequencer = MidiSystem.getSequencer();
    Transmitter transmitter = sequencer.getTransmitter();

    System.out.println("Sequencer: " + sequencer.getDeviceInfo());
    System.out.println("Transmitter: " + transmitter);
  }

  @Test
  public void testGetSynthesizer() throws MidiUnavailableException {
    Synthesizer synthesizer = MidiSystem.getSynthesizer();
    Receiver receiver = synthesizer.getReceiver();

    System.out.println("Synthesizer: " + synthesizer.getDeviceInfo());
    System.out.println("Receiver: " + receiver);
  }

  @Test
  public void testPlayNote() throws InvalidMidiDataException, MidiUnavailableException {
    // 60 = middle C, C4 (or C3 or C5)
    // https://en.wikipedia.org/wiki/Scientific_pitch_notation#Similar_systems
    int channel = 0;
    int note = 60;
    int velocity = 127; // velocity (i.e. volume); 127 = high
    ShortMessage msg = new ShortMessage();
    msg.setMessage(ShortMessage.NOTE_ON, channel, note, velocity);

    long timeStamp = -1;
    MidiDevice synthesizer = getSynthesizer("Gervill");
    synthesizer.open();
    Receiver receiver = synthesizer.getReceiver();
    receiver.send(msg, timeStamp);
    receiver.close();
  }
  
  private MidiDevice getSynthesizer(String deviceName) throws MidiUnavailableException {
     return filteredDeviceStream(deviceName)
         .filter(onClassnameEquals(SOFT_SYNTHESIZER))
         .findFirst().get();
  }

  @Test
  public void testWriteFile() throws InvalidMidiDataException, IOException {
    int volume = 0x70;
    int tick = 0;

    Sequence sequence = new Sequence(Sequence.PPQ, 24);
    Track track = sequence.createTrack();

    MetaMessage meta = new MetaMessage();
    String trackName = "First track";
    meta.setMessage(0x03, trackName.getBytes(), trackName.length());
    track.add(new MidiEvent(meta, tick));

    int notes[] = IntStream.range(60, 73).toArray();

    for (int note : notes) {
      ShortMessage on = new ShortMessage(ShortMessage.NOTE_ON, note, volume);
      track.add(new MidiEvent(on, tick));
      tick += 4;

      ShortMessage off = new ShortMessage(ShortMessage.NOTE_OFF, note, 0);
      track.add(new MidiEvent(off, tick));
      tick += 4;
    }

    MidiSystem.write(sequence, 1, MIDI_FILE);
  }

  @Test
  public void testReadFile() throws InvalidMidiDataException, IOException {
    Sequence sequence = MidiSystem.getSequence(MIDI_FILE);
    
    System.out.println("tracks: " + sequence.getTracks().length);
    System.out.println("patches: " + sequence.getPatchList().length);
    
    for (Track track : sequence.getTracks()) {
      int size = track.size();
      System.out.println("track size: "+size);
      
      IntStream.range(0, size).boxed()
        .map(track::get)
        .map(RawMessage::new)
        .map(AbstractMessage::getNoteOctave)
        .map(str -> str + " ")
        .forEach(System.out::print);
      System.out.println();
    }
  }

  @Test
  public void testMidiFileFormat() throws InvalidMidiDataException, IOException {
    MidiFileFormat format = MidiSystem.getMidiFileFormat(MIDI_FILE);

    System.out.println("bytes: " + format.getByteLength());
    System.out.println("length ms: " + format.getMicrosecondLength());
    System.out.println("divisionType: " + format.getDivisionType());
    System.out.println("resolution: " + format.getResolution());
    System.out.println("type: " + format.getType());
    System.out.println("properties: " + format.properties().size());

    format.properties().entrySet().stream()
      .map(e -> e.getValue() + "=" + e.getKey())
      .forEach(System.out::println);
  }
}

Method references kills the Factory class

06 January 2017

When writing reusable code, we often want it to be as general and flexible as possible. So layers of abstractions are added; generic types; abstract class hierarchies; and let’s not forget the Factory pattern. Joel Spolsky had a famous rant about the factory factory factory pattern, and it can get ugly in the real world as well.

One of the reasons for the often clunky factory class is that it has not been possible to pass methods and constructors. Method references in Java 8 changes that, even for constructors. They can be passed for any class or array, e.g. String::new, String[]::new. Combined with generics, the type of the newly created object can also be specified.

In the example class below, the constructor happen to take two arguments, first a String and then an int. Therefore, the BiFunction function method fits, however, it would probably be more appropriate to define a more specific functional interface, which would also make the code more readable. The return value is of the type T, which should then be the same type as the generated object. The use is demonstrated in the test method below.

The restriction with this setup is of course that the number of arguments to the constructor is fixed. We could write fixes around that as well, but that would require the general class to know something about the classes it is instantiating, which defetes the purpose. There’s always the old Factory class, though.

  class Generator<T> {

    private final BiFunction<String, Integer, T> constructor;

    Generator(BiFunction<String, Integer, T> constructor) {
      this.constructor = constructor;
    }

    T generate() {
      return constructor.apply(TEST, _123);
    }
  }

  public void testGenerateA() {
    Generator<A> genA = new Generator<>(A::new);
    A a = genA.generate();
    assertEquals(TEST + _123, a.content);
  }

Now, it can be argued that this still constitutes a factory pattern, even if an external factory class is not used. The methods of Collectors highlights this, e.g.:

    public static <T, C extends Collection<T>>
    Collector<T, ?, C> toCollection(Supplier<C> collectionFactory) {
        return new CollectorImpl<>(collectionFactory, Collection<T>::add,
                                   (r1, r2) -> { r1.addAll(r2); return r1; },
                                   CH_ID);
    }

The full code listing of the example:

GenericConstructor.java
GitHub Raw
/* Copyright rememberjava.com. Licensed under GPL 3. See http://rememberjava.com/license */
package com.rememberjava.lambda;

import static org.junit.Assert.assertEquals;

import java.util.function.BiFunction;

import org.junit.Test;

public class GenericConstructor {

  private static final String TEST = "test";
  private static final int _123 = 123;

  class Generator<T> {

    private final BiFunction<String, Integer, T> constructor;

    Generator(BiFunction<String, Integer, T> constructor) {
      this.constructor = constructor;
    }

    T generate() {
      return constructor.apply(TEST, _123);
    }
  }

  class A {
    final String content;

    A(String str, int i) {
      content = str + i;
    }
  }

  class B {
    final String str;
    final int i;

    B(String str, int i) {
      this.str = str;
      this.i = i;
    }
  }

  @Test
  public void testGenerateA() {
    Generator<A> genA = new Generator<>(A::new);
    A a = genA.generate();
    assertEquals(TEST + _123, a.content);
  }

  @Test
  public void testGenerateB() {
    Generator<B> genB = new Generator<>(B::new);
    B b = genB.generate();
    assertEquals(TEST, b.str);
    assertEquals(_123, b.i);
  }
}

Limited size queue

04 January 2017

The Java Collections Queue implementations will either grow without limit, or block if it grows beyond a given size, like the LinkedBlockingDeque. However, what if you need a non-blocking queue which drops its oldest elements? The Apache Commons CircularFifoQueue covers that. The snippet below shows typical use, with a queue size of two, and where the first element of three is dropped.

  public void testDrop() {
    CircularFifoQueue<Integer> queue = new CircularFifoQueue<>(2);

    queue.add(1);
    queue.add(2);
    queue.add(3);

    assertTrue(2 == queue.poll());
    assertTrue(3 == queue.poll());
    assertTrue(queue.isEmpty());
  }

To install the Apache Commons 4.0 library on Debian / Ubuntu:

apt-get install libcommons-collections4-java libcommons-collections4-java-doc
apt-get source libcommons-collections4-java

The relevant files will be located at:

/usr/share/java/commons-collections4.jar  
/usr/share/maven-repo/org/apache/commons/commons-collections4/4.0/commons-collections4-4.0-javadoc.jar

Often, a queue is populated on one thread, and consumed by another. In this case, the access methods have to be synchronized, as seen in this example. Both offer() and poll() methods are non-blocking, and null is returned if the queue is empty.

  public void testMultithreaded() {
    CircularFifoQueue<Integer> queue = new CircularFifoQueue<>(5);

    Thread insertThread = new Thread(() -> {
      int i = 0;
      while (true) {
        synchronized (queue) {
          queue.offer(i++);
        }
      }
    });
    insertThread.start();

    sleep(1);
    for (int i = 0; i < 10; i++) {
      synchronized (queue) {
        System.out.println("" + i + ": " + queue.poll());
      }
    }
  }

Finally, how does this queue work with Streams? In a single-thread context, there shouldn’t be a problem. However, when multithreaded it gets more tricky. The example below fails since the two threads operate on the queue concurrently, and a NoSuchElementException is often thrown. The ConcurrentLinkedQueue is thread-safe, but unbounded. Furthermore, its documentation states that “the size method is NOT a constant-time operation. Because of the asynchronous nature of these queues, determining the current number of elements requires a traversal of the elements, and so may report inaccurate results if this collection is modified during traversal”. Which means we’re back to square one.

  public void streamFail() {
    CircularFifoQueue<Integer> queue = new CircularFifoQueue<>(10);

    Thread insertThread = new Thread(() -> {
      int i = 0;
      while (true) {
        queue.offer(i++);
      }
    });
    insertThread.start();

    sleep(1000);
    System.out.println("size=" + queue.size());
    // throws NoSuchElementException
    queue.stream().forEach(System.out::println);
  }

There are a few work-arounds, mentioned in this discussion. One trick is to use the Stream.generate() method, which will loop indefinilty, and synchronize on the queue within. The problem is, that this will never stop, which might be okey depending on your application. However, you’d have to run this on a spearate thread. Alternativly, use the limit() method or a stream terminating operation (e.g. findFirst()).

  public void streamGenerate() {
    CircularFifoQueue<Integer> queue = new CircularFifoQueue<>(10);

    Thread insertThread = new Thread(() -> {
      int i = 0;
      while (true) {
        synchronized (queue) {
          queue.offer(i++);
        }
      }
    });
    insertThread.start();

    Stream.generate(() -> {
      synchronized (queue) {
        return queue.poll();
      }
    })
    // Never stops without the limit
    .limit(20)
    .forEach(System.out::println);
  }

Also worth mentioning, is the Google Guava implementation EvictingQueue. However, it it also not thread-safe.

Here’s the full listing with all test methods.

CircularFifoQueueTest.java
GitHub Raw
/* Copyright rememberjava.com. Licensed under GPL 3. See http://rememberjava.com/license */
package com.rememberjava.apache;

import static org.junit.Assert.assertTrue;

import java.util.stream.Stream;

import org.apache.commons.collections4.queue.CircularFifoQueue;
import org.junit.Test;

public class CircularFifoQueueTest {

  @Test
  public void testDrop() {
    CircularFifoQueue<Integer> queue = new CircularFifoQueue<>(2);

    queue.add(1);
    queue.add(2);
    queue.add(3);

    assertTrue(2 == queue.poll());
    assertTrue(3 == queue.poll());
    assertTrue(queue.isEmpty());
  }

  @Test
  public void testMultithreaded() {
    CircularFifoQueue<Integer> queue = new CircularFifoQueue<>(5);

    Thread insertThread = new Thread(() -> {
      int i = 0;
      while (true) {
        synchronized (queue) {
          queue.offer(i++);
        }
      }
    });
    insertThread.start();

    sleep(1);
    for (int i = 0; i < 10; i++) {
      synchronized (queue) {
        System.out.println("" + i + ": " + queue.poll());
      }
    }
  }

  @Test
  public void streamFail() {
    CircularFifoQueue<Integer> queue = new CircularFifoQueue<>(10);

    Thread insertThread = new Thread(() -> {
      int i = 0;
      while (true) {
        queue.offer(i++);
      }
    });
    insertThread.start();

    sleep(1000);
    System.out.println("size=" + queue.size());
    // throws NoSuchElementException
    queue.stream().forEach(System.out::println);
  }

  @Test
  public void streamGenerate() {
    CircularFifoQueue<Integer> queue = new CircularFifoQueue<>(10);

    Thread insertThread = new Thread(() -> {
      int i = 0;
      while (true) {
        synchronized (queue) {
          queue.offer(i++);
        }
      }
    });
    insertThread.start();

    Stream.generate(() -> {
      synchronized (queue) {
        return queue.poll();
      }
    })
    // Never stops without the limit
    .limit(20)
    .forEach(System.out::println);
  }

  private void sleep(long millis) {
    try {
      Thread.sleep(millis);
    } catch (InterruptedException e) {
      e.printStackTrace();
    }
  }
}

Sequential vs. Parallel Streams

31 December 2016

The Java 8 Streams API offers functional-style operations, and a simple way to execute such operations in parallel. To jump right into an example, the following two test methods show the difference between sequential and parallel execution of the the println() method over each of the elements of the stream.

    range = IntStream.range(1, RANGE);
  }

  @Test
  public void sequential() {
    range.forEach(PRINTLN);
  }

  @Test
  public void parallel() {
    range.parallel().forEach(PRINTLN);
  }

As is expected, the default sequential execution will print the elements of the stream, which in this example are integers from 1 to 14, in natural order. By using the parallel() method to create a parallel stream and call the same print method, the only noticeable difference is that they are printed out of order.

 --- sequential ---      --- parallel ---
1                       9
2                       12
3                       5
4                       4
5                       13
6                       7
7                       6
8                       11
9                       10
10                      1
11                      2
12                      14
13                      3
14                      8

To see how the parallel stream behaves, the last test applies a different method to each of the elements of the stream. In this example, the time spent by the work() method on each element is linearly proportional to its value. That is, for the element of value 1 it spends 100ms, for 2 its 200 ms and so on. The “work” it does, is simply to sleep for intervals of 100 ms, and the rest of the code is dedicated to printing and formatting the table below. However, it serves the purpose of demonstrating how parallel execution behaves, and how it relates to the underlying CPU(s).

    range.parallel().forEach(this::work);

In the result table, each element of the stream is represented by a row. Each row shows the value of the element, the time-slots the work() method was executing, and to the right the time in milliseconds when work started and finished. Each 100 ms slot is represented by a hash. As can be seen, the first row was element 9, thus it marked off nine slots, and the starting time was at 6 ms, and finish at 909 ms.

Furthermore, since this was run on a machine with four CPU cores, the stream will execute four calls in parallel. This can be seen by both the hashes and the start times of the first four rows. Next, when element 2 (fourth row) finishes at 207 ms, a new element is immediately started (element 3, fifth row).

In this example, the total number of 100 ms “units of work” can be found by the formula for the triangular number where n = 14, or 14 * (14 + 1) / 2 = 105. Meaning that, sequential execution would have taken 10.5 seconds, while four parallel CPUs managed in 3 seconds.

In the second table below, the same code is executed on a dual core CPU, and it is clear that now only two methods execute in parallel. That will of course lead to a longer overall runtime, of about 5.4 seconds for this example. This could lead to a discussion on task and scheduling optimisation, however it goes beyond this article, and what is possible with the simple parallel Stream construct.

 --- parallelWork ---
CPU count: 4

    0000000000111111111122222222223333
    0123456789012345678901234567890123
 9:  #########                          [   6 -  909]
13:  #############                      [   6 - 1312]
 5:  #####                              [   6 -  507]
 2:  ##                                 [   6 -  207]
 3:    ###                              [ 207 -  507]
 4:       ####                          [ 507 -  909]
 1:       #                             [ 507 -  608]
 7:        #######                      [ 609 - 1312]
10:           ##########                [ 911 - 1915]
12:           ############              [ 911 - 2116]
14:               ##############        [1313 - 2719]
 6:               ######                [1313 - 1915]
 8:                     ########        [1915 - 2719]
11:                     ###########     [1915 - 3021]
    0123456789012345678901234567890123
    0000000000111111111122222222223333
 --- parallelWork ---
CPU count: 2

    00000000001111111111222222222233333333334444444444555555555
    01234567890123456789012345678901234567890123456789012345678
 8:  ########                                                    [  23 -  825]
 4:  ####                                                        [  23 -  424]
 5:      #####                                                   [ 424 -  926]
 9:          #########                                           [ 826 - 1727]
 6:           ######                                             [ 926 - 1527]
 7:                 #######                                      [1527 - 2228]
10:                   ##########                                 [1727 - 2729]
 1:                        #                                     [2229 - 2329]
 2:                         ##                                   [2329 - 2529]
 3:                           ###                                [2530 - 2830]
13:                             #############                    [2729 - 4031]
11:                              ###########                     [2831 - 3932]
12:                                         ############         [3933 - 5134]
14:                                          ##############      [4031 - 5433]
    01234567890123456789012345678901234567890123456789012345678
    00000000001111111111222222222233333333334444444444555555555

The full code list is here.

ParallelCount.java
GitHub Raw
/* Copyright rememberjava.com. Licensed under GPL 3. See http://rememberjava.com/license */
package com.rememberjava.lambda;

import java.time.Duration;
import java.time.Instant;
import java.time.temporal.TemporalAmount;
import java.util.Arrays;
import java.util.List;
import java.util.Vector;
import java.util.function.IntConsumer;
import java.util.stream.IntStream;

import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestName;

public class ParallelCount {

  @Rule
  public TestName name = new TestName();

  private static final IntConsumer PRINTLN = System.out::println;

  private static final int RANGE = 15;

  private int axisWidth;

  private static final int WORK_UNIT_MILLIS = 100;

  private List<StringBuilder> results = new Vector<>(RANGE);

  private TemporalAmount start;

  private IntStream range;

  @Before
  public void setup() {
    String methodName = name.getMethodName().split("\\[")[0];
    System.out.println("\n --- " + methodName + " ---");

    int cpus = Runtime.getRuntime().availableProcessors();
    int workUnits = RANGE * (RANGE + 1) / 2;
    axisWidth = (int) ((workUnits / cpus) * 1.3);

    start = Duration.between(Instant.EPOCH, Instant.now());

    range = IntStream.range(1, RANGE);
  }

  @Test
  public void sequential() {
    range.forEach(PRINTLN);
  }

  @Test
  public void parallel() {
    range.parallel().forEach(PRINTLN);
  }

  // TODO: Fix index failure
  //@Test
  public void disabled_parallelWork() {
    System.out.printf("CPU count: %d\n\n",
        Runtime.getRuntime().availableProcessors());
    printAxis(true);

    range.parallel().forEach(this::work);

    results.stream().map(sb -> sb.toString()).forEach(System.out::println);
    printAxis(false);
  }

  void work(int units) {
    StringBuilder result = new StringBuilder();
    results.add(result);

    char[] timestampSlots = new char[axisWidth];
    Arrays.fill(timestampSlots, ' ');

    long startMillis = getNowMillis();

    try {
      for (int i = 0; i < units; i++) {
        Thread.sleep(WORK_UNIT_MILLIS);

        int nowSlot = (int) (getNowMillis() / WORK_UNIT_MILLIS);
        timestampSlots[nowSlot] = '#';
      }
    } catch (InterruptedException e) { // ignore
    }

    long endMillis = getNowMillis();

    result.append(String.format("%2d: %s [%4d - %4d]",
        units, new String(timestampSlots), startMillis, endMillis));
  }

  private void printAxis(boolean header) {
    String row1 = "";
    String row2 = "";
    for (int i = 0; i < axisWidth - 1; i++) {
      row1 += i / 10;
      row2 += i % 10;
    }

    System.out.println("    " + (header ? row1 : row2));
    System.out.println("    " + (header ? row2 : row1));
  }

  private long getNowMillis() {
    return Instant.now().minus(start).toEpochMilli();
  }
}

TrayIcon Example

17 December 2016

Tray Icon

Since Java 6, adding a system tray icon has been straight forward. The two main classes involved are SystemTray and TrayIcon. Various OS might render and operate the icon differently. Typically, there is a status message on hover, a short-cut action on left click, and possibly a menu on right click. The TrayIcon supports all this. In the example code below, an 16x16 pixel PNG is used, and auto-scaled up to about 24 pixels, which is what I’ve configured by XFCE panel.

  private void initIcon() throws AWTException {
    if (!SystemTray.isSupported()) {
      System.err.println("System tray not supported.");
      return;
    }

    Image image = Toolkit.getDefaultToolkit().getImage(
        "com/rememberjava/ui/favicon_16.png");

    TrayIcon trayIcon = new TrayIcon(image, "RJ Tray Demo", popup);
    trayIcon.setImageAutoSize(true);

    SystemTray tray = SystemTray.getSystemTray();
    tray.add(trayIcon);
  }

Here’s the full example class listing.

TestTrayIcon.java
GitHub Raw
/* Copyright rememberjava.com. Licensed under GPL 3. See http://rememberjava.com/license */
package com.rememberjava.ui;

import java.awt.AWTException;
import java.awt.Image;
import java.awt.MenuItem;
import java.awt.PopupMenu;
import java.awt.SystemTray;
import java.awt.Toolkit;
import java.awt.TrayIcon;
import java.awt.event.ActionEvent;

public class TestTrayIcon {

  private PopupMenu popup;

  public static void main(String[] args) throws Exception {
    TestTrayIcon test = new TestTrayIcon();
    test.initMenu();
    test.initIcon();
  }

  private void initMenu() {
    popup = new PopupMenu();

    addMenuItem("item1");
    addMenuItem("item2");
  }

  private void addMenuItem(String label) {
    MenuItem item = new MenuItem(label);
    item.addActionListener(this::click);
    popup.add(item);
  }

  private void click(ActionEvent e) {
    System.out.println("Clicked: " + e.getSource());
  }

  private void initIcon() throws AWTException {
    if (!SystemTray.isSupported()) {
      System.err.println("System tray not supported.");
      return;
    }

    Image image = Toolkit.getDefaultToolkit().getImage(
        "com/rememberjava/ui/favicon_16.png");

    TrayIcon trayIcon = new TrayIcon(image, "RJ Tray Demo", popup);
    trayIcon.setImageAutoSize(true);

    SystemTray tray = SystemTray.getSystemTray();
    tray.add(trayIcon);
  }
}

A simple stupid calculator

12 December 2016

Calculator

This post includes the UI (Swing) for a very simple calculator. There’s not much to say about the code, except for the rather stupid way it handles the calculator operation itself: Using a JavaScript engine! The ScriptEngineManager and the internal NashornScriptEngineFactory JavaScript implementation have been around since Java 6 and 8 receptively. It makes it easy to execute a string as snippet of code, as seen below.

Here the model for the display of the calculator is just a plain string. That string is then evaluated as a line of JavaScript, and the output is returned and put back into the “model”.

  {
    NashornScriptEngineFactory factory = new NashornScriptEngineFactory();
    engine = factory.getScriptEngine();
  }

  ...
  String model = "1 + 1";

  void calculate() {
    try {
      Object eval = engine.eval(model);
      model = eval.toString();
    } catch (ScriptException e) {
      // If the expression was invalid,
      // don't modify the calculator display.
    }
  }

In terms of the calculator functionality, this implementation is very simple, but therfore also limited. A more common way of implementing this would be through some object expression representation which can be evaluated. However, the Script Engine implementation has the benefit of supporting functionality which is not even implemented in the UI, like brackets or other operators like power-to (^). It works nicely has a prototype and quick mock, so maybe not so stupid after all.

To be continued…

For the full source of this first version, see here.

Tutorialspoint - compile and run code online

05 December 2016

Over at Tutorialspoint.com they have created a superb well of study material and tools. Most impressive are the online compiler and runtime terminals, dubbed Coding Ground, for a large selection of popular and obscure languages and tools. Here’s an example Java project, based on the previous article on word counting. Click Compile followed by Execute and the output from the snippet will show in the terminal. The data file “words” is included in the project. (Note that these terminal seems to run best in Chrome based browsers, and fails to load in Firefox).

Also worth checking out is their long list of tutorials for everything from Java; other programming languages; to sports and “soft skills”. Especially with the programming tutorials, their embedded online compiler & terminal feels very slick, as seen below.

My personal site fades in comparison, however, my goal is not to offer complete university courses. Remember Java will remain focused on exploration and examples of interesting aspects of the Java language.

Demo project

Tutorial

Newer posts Older posts