Sending Logs to the Cloud

Logging is a key tool for the application developer, with many considerations such as what detail to log, logging level, etc. Here we’ll consider where to log and use Hazelcast as a log destination for Slfj4, the standard logging framework for Java. Specifically, we’ll use an IMap as our log store, and store it on the cloud. An IMap makes it easier to work with than logging to the filesystem, as there might be many host machines involved in your application, so log collation is solved by hosting the logs in Hazelcast.

Why an IMap ?

An IMap is essentially a random access key value store. Key hashing means that consecutive keys are probably not stored adjacent to each other, allowing for easy rebalancing as data grows. What matters is how we query the log store. Most likely we will be doing range scans, all the logs for the last few minutes perhaps. This would require a sort since the data is not stored sorted. So, by using an IMap we have gained storage scalability at the expense of query speed, which seems a reasonable trade-off.

The log structure

For querying, we define the IMap content as:

        String mapping = "CREATE OR REPLACE MAPPING \"" + logMap.getName() + "\""
                + " ("
                + "    \"socketAddress\" VARCHAR EXTERNAL NAME \"__key.socketAddress\","
                + "    \"timestamp\" BIGINT EXTERNAL NAME \"__key.timestamp\","
                + "    \"level\" VARCHAR EXTERNAL NAME \"this.level\","
                + "    \"message\" VARCHAR EXTERNAL NAME \"this.message\","
                + "    \"threadName\" VARCHAR EXTERNAL NAME \"this.threadName\","
                + "    \"loggerName\" VARCHAR EXTERNAL NAME \"this.loggerName\""
                + " )"
                + " TYPE IMap "
                + " OPTIONS ( "
                + " 'keyFormat' = 'json-flat',"
                + " 'valueFormat' = 'json-flat'"
                + " )";

The key is JSON composed from the IP address of the process that writes to the log, plus a timestamp. The value is also JSON, the more familiar logging fields of log level, message and so on. This blog post is just demonstrating the concept. For real use we’d need to expand it to distinguish between multiple processes on the same host, and perhaps using a JSON Array to capture multiple messages produced with the same timestamp.

Show Me the Code

Our application code does this:

       HazelcastInstance hazelcastClient = HazelcastClient.newHazelcastClient(clientConfig);
       org.slf4j.Logger logger = IMapLoggerFactory.getLogger(Application.class);
       logger.info("hello world");
       hazelcastClient.shutdown();

We create a connection to a Hazelcast cluster using some client configuration. This may be to Hazelcast Viridian or an already active cluster. Then we obtain a SLF4J Logger that uses Hazelcast as a destination. Then we log a message and shutdown. Crucially, the rest of the application may have nothing to do with Hazelcast. What we’re using the Hazelcast client for here is to connect to the logging store (the Hazelcast cluster), so as a minimum it’s a write-only client.

Show Me the Code in Detail

Two classes do the work.

IMapLoggerFactory

IMapLoggerFactory returns IMapLogger instances, implementing the ILoggerFactory interface. It looks like:

public class IMapLoggerFactory implements ILoggerFactory {

    private static IMap logMap;
    private static String socketAddress;
    private static Level level = Level.INFO;

    public static synchronized Logger getLogger(Class klass) {
        if (logMap == null) {
            Iterator serverIterator = Hazelcast.getAllHazelcastInstances().iterator();
            Iterator clientIterator = HazelcastClient.getAllHazelcastClients().iterator();

            HazelcastInstance hazelcastInstance = serverIterator.hasNext() ? serverIterator.next() : clientIterator.next();

            socketAddress = hazelcastInstance.getLocalEndpoint().getSocketAddress().toString();
            logMap = hazelcastInstance.getMap("log");
        }
        return new IMapLogger(klass.getName(), logMap, socketAddress, level);
    }

    public static void setLevel(Level arg0) {
        level = arg0;
    }

    @Override
    public Logger getLogger(String name) {
        return LoggerFactory.getLogger(name);
    }

}

Most of the work is in finding the current Hazelcast instance.

We use Hazelcast.getAllHazelcastInstances() to find all server instances in this JVM, and HazelcastClient.getAllHazelcastClients( to find all clients.

We assume this JVM has exactly one Hazelcast instance. If this isn’t going to be true, you’d need to adjust the implementation.

From that Hazelcast instance, we create a logger passing it the IMap to log to.

IMapLogger

IMapLogger implements the Logger interface, which has more than 60 methods.

public class IMapLogger implements Logger {

    private final String name;
    private final IMap logMap;
    private final String socketAddress;
    private final Level level;

    public IMapLogger(String arg0, IMap arg1, String arg2, Level arg3) {
        this.name = arg0;
        this.logMap = arg1;
        this.socketAddress = arg2;
        this.level = arg3;
    }

    private void saveToHazelcast(Level lvl, String msg) {
        StringBuffer keyStringBuffer = new StringBuffer();
        keyStringBuffer.append("{");
        keyStringBuffer.append(" \"socketAddress\" : \"" + this.socketAddress + "\"");
        keyStringBuffer.append(", \"timestamp\" : " + System.currentTimeMillis());
        keyStringBuffer.append("}");

        StringBuffer valueStringBuffer = new StringBuffer();
        valueStringBuffer.append("{");
        valueStringBuffer.append(" \"loggerName\" : \"" + this.name + "\"");
        valueStringBuffer.append(", \"threadName\" : \""
                + Thread.currentThread().getName() + "\"");
        valueStringBuffer.append(", \"level\" : \"" + lvl + "\"");
        valueStringBuffer.append(", \"message\" : \"" + msg + "\"");
        valueStringBuffer.append("}");

        this.logMap.set(new HazelcastJsonValue(keyStringBuffer.toString()),
                new HazelcastJsonValue(valueStringBuffer.toString()));
    }

The main method added is saveToHazelcast. This turns the logging detail into a JSON key and a JSON value to insert into the logging IMap.

For “INFO” logging, we have this:

    @Override
    public boolean isInfoEnabled() {
        return this.level.toInt() <= Level.INFO.toInt();
    }

    @Override
    public void info(String msg) {
        if (this.isInfoEnabled()) {
            this.saveToHazelcast(Level.INFO, msg);
        }
    }

So when we do logger.info(“hello world”) it invokes our method if INFO logging is on. The methods for WARN, DEBUG etc logging are omitted for brevity. They follow a similar pattern.

Querying

From the Management Center or a client, we can query our logs.

Summary

So there you have it, logs from wherever can be saved to the cloud for later inspection.

If you’re interested in a fuller example, see here.

To avoid running out of space, you might want to configure the IMap for eviction and/or expiry.

IMPORTANT NOTE

You should absolutely consider security here. Confidential data may appear in logs, and you don’t want accidental or malicious connections.