OpenTSDB

I’ve been hearing good things about OpenTSDB   for some time and I’ve been meaning to take it for a spin.

Well I’ve finally gotten around to it.

First impressions:

  • Pretty damn easy to set up. Considering the technology behind this.

The instructions are easy to follow and the build.sh script does all the hard work.  I’m running a single HBase server. Not a cluster. HBase is the Hadoop database.  Wooo Big Data. I’m running it on an AWS t1.micro (small Big Data). Just evaluating it at the moment.

  • UI probably designed by an Engineer.

Functional but not so pretty.  Hey I’m not saying I can do better!  Frontend is not something I’m good at.  You should see my horrible HTML.  It would make your eyes bleed.  Blink tag is still cool yeah?  The thing that is kinda annoying is that it seems to use the time zone from my browser, which is not the time zone of my servers (UTC).  Graphite’s UI took a long time for me to warm to.  Now I’d like to see something like that for OpenTSDB.  I believe there are some projects along these lines but I haven’t investigated.

  • NoSQL to be Web Scale.  Ok I can push billions of data points into HBase and never lose data like I do in Graphite (whisper) or Ganglia (RRD).  Cool.

As I understand it, there’s no real difference between the concept of a Metric, and a key/value pair Tag.  It’s all just data in NoSQL magic land.

This flexibility is proving difficult for me to work out how to get data in.  I’m working on a project in my hack days at 99designs to suck metrics in from a variety of services and spit them out into a variety of backend systems.   Ganglia and Graphite so far and now OpenTSDB.  Just to see where it leads.   It’s open source and on github here:  https://github.com/99designs/vacuumetrix  But I’m not sure how to use tags yet.  There’s some talk on the mailing list about changing how this works, so I’m not going to fret too much.   https://groups.google.com/d/msg/opentsdb/llAVqkKqFPw/_vQbf1MSX6sJ

  • Downsampling.  Ok this is really useful, and is something the likes of Ganglia and Graphite don’t do so well.  Sometimes to see what is going on in the data you want to downsample.
  • Aggregation is the default.  OpenTSDB is designed to handle huge abouts of data. So aggregating metrics is naturally on.
  • You have to manually register each new metric.
    e.g.  #tsdb mkmetric newrelic  This is a bit sucky.  Is there a way to do it automatically?
  • tcollector I’ve no idea if this is working or not.  Admittedly I’ve only been looking at OpenTSDB for a couple of hours.  It’s not immediately clear how this works or what the metrics it may or may not be injecting are called.  I have not yet looked at the source.  I just tried to run something called startstop?!?  If I have a few moments I’ll investigate further.

In conclusion.  The prospect of not throwing away data is attractive.  I will keep playing with this tool, get some real data in and see if it’s useful.

I recommend watching this video about OpenTSDB.

4 Comments

Filed under Uncategorized

4 responses to “OpenTSDB

  1. Luke R

    option you want is: –auto-metric
    As for getting data into opentsdb you just need to send the right format data to the port opentsdb is listening on.
    The examples use netcat with the input format:
    put metric timestamp value tags

  2. Ah, thanks Luke. –auto-metric is exactly what I wanted.

  3. found any good frontends for opentsdb? missing something with a tree structure (like graphite, cacti etc.).. I tried out tsdash from facebook – but it won’t work with hbase-0.94..

  4. Oh that’s a shame. Looks promising. https://github.com/facebook/tsdash/issues/7 Yes, I missed the tree structure for browsing metrics too!

    No I’m not using any alternative frontends at the moment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s