All the lias

Hit by IPv6: Tomcat does not start on Debian

Al — Mon, 15 Mar 2010 10:15:15 +0000

Finally I got hit with this one. Its a Debian + Java + IPv6 misunderstanding that may take time to figure out. The fix with -Djava.net.preferIPv4Stack=true in the Java start command is rather simple. For Tomcat, you can put it into the variable JAVA_OPTS of your environment.

export JAVA_OPTS=' -Djava.net.preferIPv4Stack=true'

This will fix your Java app only; if you dont have other software running against the kernel stack, you may disable this behaviour altogehter. Following this advice you can issue

sudo sed -i 's/net.ipv6.bindv6only\ =\ 1/net.ipv6.bindv6only\ =\ 0/' \
/etc/sysctl.d/bindv6only.conf && sudo invoke-rc.d procps restart

(It may crash some running application though; better restart your system. You have to restart your java servers anyhow, as they need to rebind to the interfaces). This latter change allows you also, to get back using the libtcnative-1 package that comes with the recent debians.

HbaseExplorer gets own Blog

Al — Sun, 14 Mar 2010 20:53:59 +0000

All Information about the HBaseExplorer moved to its own blog!

2nd Munich OpenHUG

Al — Fri, 26 Feb 2010 12:52:32 +0000

Christoph Rupp started with a introduction to his embeddable HamsterDB, a nice and light alternative for BerkeleyDB. Beside a comprehensive feature set, he impressed with strong quality rules that are applied to each release. A embedded fast DB is certainly a different world than the massive scaling we do with Hadoop & Co, but why not imagine the HBase Shards using a really fast embedded piece of C-Code?

Afterwards we had some good discussions in different directions, spanning from Hadoop* up to all kinds of cloud challenges.

This was our second meeting and I still assume there are more data-fighters out there in the munich area. Spread the word!

Looking forward to he next Open Hadoop Users Group Meeting, probably mid-May 2010.

Where to put my persistence layer?

Al — Tue, 19 Jan 2010 20:56:54 +0000

After you play a while with the put()’s and get()’s of your Hbase client, you’ll probably start to think about how to organize the mess.

When a classic DAO…

This classic approach with one backend database ensures that the client code needs no knowledge of the database and also not of the actual persistance strategies in the Persistance Layer (which was called DataAccessObject some years ago). Modern software can even drive several database models just by different configurations. Proper interfaces allow a independend testing of all layers.

The Database itself ensures the data consistency and has a strong API (SQL).

All thats nice, but now you want to manage a lot of data, and you may choose HBase to do this.

…gets distributed

When using a distributed, shared-nothing database, there is -by definition- no single point that manages the persistance strategies, such as

Ensuring Consistency (was the job of the RDBMS)
Transactions (RDBMS)
Maintain Indexes (RDBMS and your admin)
CRUD into/from several Tables (classically a job of the Persistance Layer)
Caches and Buffers to heal short Database Hickups
Handle Security (Code or Database)

For a first attempt, just let a new persistance layer for the distributed database do all this:

(In a shared-nothing Database, as Shard holds a fragment of your data row set. A Master typically somehow knows of everything, but has no data.)

But how about consistency? We would have to update all clients in the same time to make sure, everybody properly maintains the index, adheres to the new security rules etc. Also, if we have more than one application, this gets complicated. Imagine 24×7.

So changes on the persistance layer require a immediated distribution to all clients that use it. This is new and quite different to classical RDBMS: “Lets put an index” is not so simple anymore. There are several options to get the persistance code distributed:

A	shutdown clients, redeploy, restart	Normally not possible, also there is the risk of forgetting a client
B	build some schema version checking, let clients check the DAO Version for any access, Reload the DAO Code dynamically	While loading code dynamically is really cool, your QA Department will probably not like it. You need good security measures as well….
C	Have a additional layer of servers that act as DAO Layer	This seems to promise a solution to many problems at the cost of that additional servers.

A separate set of DAO servers

So if you have the money for some additional servers, this is the way to go. It offers solutions to all the problems mentioned above.

Performance might be a problem. The beauty of shared-nothing comes from the independend life that each thread in your business logic can live. If you query Google, you might -in that moment- well have 20 computers available, just for your single request. This additional layer should scale at the same rate than the other IO streams of your application, possibly in a one-to-one relation as shown in the picture above. If you have some reasonaable work to do in your DAO Layer (such as encryping some fields, or calculating hashes for indexes), this computing power is not only additional cost, it frees your business layer from that.

So you have your separate DAO. How to update them now??? Restarting all the same time is also a short downtime. So here you are challanged to write code that allows you to do the mentioned things at runtime, such as changing permission settings or adding index rules. After all, these servers can also hold database maintainance code as well.

You might want to use a load-balancing between the Client and the DAO Nodes, which gives you the additional benefit of scaling and replacing nodes at runtime. The DAO Nodes may well buffer calls or run them in a multithreaded fashion, to give better reponse time from the database to the clients. A Firewall can offer addtional safety in your datacenter.

With all this freedom, dont forget that such a design does still not offer many things you may have got used to from traditional RDBM’s – unless you’d put them into your DAO Layer code yourself (if you are a hard-core database expert). But you may not need many of these things – and if the scaling benefits play out the potential loss of precision and accuracy – the data storage will never limit your business anymore.

–Al

JSP Wiki

Al — Mon, 18 Jan 2010 08:49:17 +0000

Some days ago I used the Forum Plugin for JSP Wiki I wrote some time ago. As one of the few Java based Wiki’s around, its relativly easy to customize and to embed, even in complex environments. Janne maintains this Software since many years, so I want to spread the word about it a bit here.

You can use it as a simple content management, as I did in this or that project.

hbasexplorer on sourceforge

Al — Fri, 15 Jan 2010 22:27:07 +0000

Dont you know about Hbase ? Ist something that finishes with the Idea of traditional databases. Altough it does not completely deny the good things that RDBMS invented, it gets you to a new level of data storage. If you are willing to change. The idea is well (but somewhat complicated) described in Google’s papers, great folks followed that idea and put it into a public, really working piece of software.

Shared-nothing rules. Try it! Search something at Google. Or Bing.

I want to make it a bit more handy by starting a tool that can visualize the data in timestamp order, providing inter-linking between the data and more. I think this is the real revolutionary idea of BigTable: dont look at the data only as a snapshot in time, but with the idea of history (and future) in mind.