Simple python script for generating Cassandra initial tokens

When using a RandomPartitioner, it is recommended that you specify the initial tokens. On the Cassandra Operations wiki page, it says:

Using a strong hash function means RandomPartitioner keys will, on average, be evenly spread across the Token space, but you can still have imbalances if your Tokens do not divide up the range evenly, so you should specify InitialToken to your first nodes as i * (2**127 / N) for i = 0 .. N-1. In Cassandra 0.7, you should specify initial_token in cassandra.yaml.

Here is a simple python script for generating them:

#! /usr/bin/python
import sys
if (len(sys.argv) > 1):
    num=int(sys.argv[1])
else:
    num=int(raw_input("How many nodes are in your cluster? "))
for i in range(0, num):
    print 'node %d: %d' % (i, (i*(2**127)/num))

So it will take either a command-line arg for the number of nodes or will ask if none is given. For three nodes, it will give the following output:

node 0: 0
node 1: 56713727820156410577229101238628035242
node 2: 113427455640312821154458202477256070485

This post was adapted from this, just updated the script and corrected the formula.

Installing Sun Java on Lucid Lynx

First I had to install python-software-properties to be able to run add-apt-repository:

sudo apt-get install python-software-properties

Then I add the repo that has Sun's jdk:

sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
sudo aptitude update
sudo aptitude install sun-java6-jdk

Then, if there are multiple jdk alternatives on the system, choose which one you want with:

sudo update-alternatives --config java

Large-scale Storage/Computation at Google

Stu Hood pointed me to an interesting keynote today done by Jeffrey Dean, a Fellow at Google. It's part of the ACM Symposium on Cloud Computing. In it, he talks about the current large scale storage and computation infrastructure used at Google. It starts off kind of slow but picks up (for me) after he talks a bit about MapReduce.

The presentation with slides is available here (silverlight required)

Some interesting bits to me:

• He talked about several patterns for distributed systems that they have found useful.
  
• Google currently MapReduces through about an exabyte of data per month
• Interesting example of how they use MapReduce - to return the relevant map tiles in Google Maps for a given query
• He pointed out that they have Service Clusters of BigTable so that each group doesn't have to maintain their own - this relates to what Stu and I are doing at Rackspace - creating multi-tenant Hadoop and Cassandra clusters for similar reasons
• They use ranged distribution of keys for BigTable, saying that consistent hashing is good in some ways, but they wanted to be able to have locality of key sequences.
• He talked about something I've been looking at recently - how to do custom multi-datacenter replication by table (or for Cassandra by keyspace).

Presentation on Cassandra+Hadoop

Last night I gave a presentation at the Austin Hadoop User Group about Cassandra + Hadoop. It was a great group of people in this relatively new user group here, probably around 20-30 people were there.

My slides are available in keynote form on slideshare - linked here:

Cassandra+Hadoop

View more presentations from Jeremy Hanna.

Steve Watt from IBM's BigSheets and Emerging Technologies team pointed out that Cassandra has an edge over the native Hadoop technologies in that you can query output in Cassandra immediately. Using just Hadoop, especially HDFS for the output, you have to export the MapReduce output to another system if you want to do any kind of reporting. That can be a significant extra step.

Stu Hood also pointed out that even though it is possible to run MapReduce over data in Cassandra, HDFS and HBase are built to stream large chunks of data. So Cassandra will be slower from that perspective at this point. Work can be done to optimize that though. I think no matter what, you're choosing your data store for a variety of reasons - if your data fits better in Cassandra, now you have an option of running MapReduce directly over it. I think that's a significant advance.

Lots of thanks to Stu Hood as well as Jeff Hammerbacher (on #hadoop IRC) for help on some of the details.

With this done with, it's back to doing what I can to help with Cassandra dev and looking forward to the Hadoop Summit at the end of the month.

Facebook and Privacy

I recently decided to deactivate my Facebook account. I did that based on several reports of Facebook disregarding the privacy of users in order to further monetize their platform. To be sure, they have a fantastic platform and I've liked being able to connect with people I haven't seen in a long time. However, hosting user data comes with the responsibility of keeping the trust of the user. To me, for now they've broken that trust. Anyway, I just thought I would post what I sent to Facebook when I deactivated my account:

According to several reports lately from the EFF, Wired, and several online publications, Facebook has consistently changed privacy terms out from underneath users. Part of the reason that I felt safe on Facebook and not in other networks was that I trusted Facebook to some extent. Based on these new changes and the seeming disregard for its users, I would rather not support Facebook any longer. Thank you for the remarkable service, but right now I don't feel that Facebook is trustworthy. They seem like they will do anything in order to further monetize the network/platform, including compromise the trust of its users. It's an unfortunately short-sighted gamble and I hope you will reconsider.

See:
http://www.wired.com/epicenter/2010/05/facebook-rogue/
http://www.eff.org/deeplinks/2010/05/things-you-need-know-about-facebook
http://www.eff.org/deeplinks/2010/04/facebook-timeline
http://www.pcworld.com/article/195888/facebooks_antiprivacy_backlash_gains_ground.html

NOSQL

As I've started getting up to speed at my new job at Rackspace down here in Texas, I've come into a new world called NoSQL. NoSQL is a term that Eric Evans re-coined relatively recently and he's since clarified that to mean Not only SQL. It's a term that kind of describes a set of distributed databases that have some similar properties.

Some of the suspects include Google's BigTable, Hadoop's HBase, Amazon's Dynamo, Apache's Cassandra, CouchDB, MongoDB, Voldemort, and others.

It seems to be based on the notion that if you have really, really, really large data sets, you run into some boundaries with the limits that a relational database imposes with ACID properties, transactions, and the unattainable triforce of Consistency, Availability, and Partition-tolerance (from the CAP Theorem). Jonathan Ellis blogged about deciding whether you should consider a NoSQL solution here.

So I've started drinking from a firehose of sources to try to understand more about them. We've been looking heavily into pieces of the Hadoop project for its distributed filesystem and Map/Reduce implementation (not exactly NoSQL but siblings to HBase), as well as the Cassandra project because of how it brings together useful features of BigTable and Dynamo and allows for completely horizontal scaling - no single point of failure.

More about the subject:
http://www.royans.net/arch - a blog about scalable web architectures, often talking about big data and NoSQL
http://nosql.mypopescu.com - a blog called myNoSQL that deals with all things NoSQL

list comprehensions in python

One of my favorite features of python is a functional language feature that python itself borrowed - list comprehensions.

I just think it is wonderfully elegant if a language can do something like this:

Example 1:

lines = ['now is the time\r\n']
lines.append(' for all good men ')
lines.append(' to come to the aid of their country\n')

# Does not modify list, returns a new list
lines = [line.strip() for line in lines]

print lines

>>>['now is the time', 'for all good men', 'to come to the aid of their country']

Example 2:

lines = ['<act> is the next act']
lines.append('performing at our show')
lines.append('please give <act> a big round of applause')

print [line.replace('<act>', 'Go Dog Go') for line in lines]
print [line.replace('<act>', 'The Beatles') for line in lines]

>>>['Go Dog Go is the next act', 'performing at our show', 'please give Go Dog Go a big round of applause']

>>>['The Beatles is the next act', 'performing at our show', 'please give The Beatles a big round of applause']

You can do simple operations like strip and replace or even an in place lambda on every element of a list and return that list... all in one line.

I love Python and its functional cousin languages.

hibernate console in intellij idea 9

I was pleasantly surprised to find better hibernate support in intellij idea 9, which was recently released.

They had hibernate support in 8.x, including a console, where you could run queries against your data using the mappings you had configured. However in version 9, they added the ability to use named parameters with associated values. So in essence, you can paste in a hql query and it autodetects the named parameters, e.g. :username. That pops up on the right pane as a named paraemeter. You just double click that, set its value, and you can run the query.

intellij has had support for this in their jdbc console in the past, but it's very handy now with the hibernate console. It removes one step from having to debug queries - you no longer have to use just the sql output that hibernate outputs and then piece queries back together and then try to guess where the disconnect was :).

See intellij feature request:
http://youtrack.jetbrains.net/issue/IDEADEV-41129

Related to hibernate console, don't forget to have the ehcache.jar in your module's classpath if you want to use the hibernate console. You can find the jar in the basic core download of hibernate - in the lib/optional/ehcache directory of the bundle. The console requires the secondary cache and will give you odd secondary cache errors if you don't have it in the path.

See this about loading ehcache:
http://youtrack.jetbrains.net/issue/IDEA-21914

Custom hotkeys in IntelliJ IDEA

Three custom hotkeys I use quite frequently in IntelliJ IDEA:

alt-shift-L - Compare with latest repository version
creates a diff from your copy to the latest version of the current file from the repository

alt-shift-H - Show History
shows the version history of the file with who modified it, revision number, and comments

alt-shift-A - Annotate
annotates the current file with the revision number and who modified each line last - I love this one.

To create custom hotkeys, go to File->Settings->Keymap.

You can find those mappings under Version Control Systems.

Scrum?

So I've been on several teams that refer to themselves as agile. They do scrum meetings each day and get updates from individuals on the team. Presumably, the meeting is for coordination of effort among the individuals.

Scrum seems to work better in some groups - finding holes in requirements, promoting discussion about a data model, general communication to make sure everyone can deliver for the next iteration.

Sound good? Sound normal? Sound effective?

Well I've wondered lately about the cumulative time from all the individuals in the room - that's a lot of work time. That's a lot of disruption. That's a lot of "I'm working on bugs" on some days.

Then today I was reading a passage in the book Peopleware that warns of a balance.

The ultimate management sin is wasting people's time.
...
When you convoke a meeting with n people present, the normal presumption is that all those in the room are there because they need to interact with each other in order to come to certain conclusions. When, instead, the participants take turns interacting with one key figure, the expected rationale for assembling the whole group is missing; the boss might just as well have interacted separately with each of the subordinates without obliging the others to listen in.

He goes on to say that some ceremonial meetings are necessary, for project milestones, when new people come on, celebrating a release, etc. However, the authors in the same section of the book say:

A real working meeting is called when there is a real reason for all the people invited to think through some matter together. The purpose of the meeting is to reach consensus. Such a meeting is, almost by definition, an ad hoc affair. Ad hoc implies that the meeting is unlikely to be regularly scheduled. Any regular get-together is therefore somewhat suspect as likely to have a ceremonial purpose rather than a focused goal of consensus. The weekly status meeting is an obvious example. Though its goal may seem to be status reporting, its real intent is status confirmation. And it's not the status of the work, but the status of the boss.

Weekly status meetings?!? What about a daily status meeting?

Now I'm not saying that scrum is always a waste of everyone's time. However, I wonder if we in the world of agile are missing the point sometimes and ceremony trumps getting work done. I wonder if many of the same things could be accomplished by having a common work area online, like a campfire chat room or an IRC channel for work discussions. I thought a recent interview (links to page 2) with Jason Fried of 37 Signals was interesting - his take on meetings.

The excerpts from Peopleware come from chapter 33: "The Ultimate Management Sin Is ..." It goes on to talk about all sorts of ways to waste people's time.

I just thought it was interesting to contrast the need in agile for a scrum-like meeting with the need for uninterrupted work time. I think it just inspires thought about whether or not a given meeting, particularly a regularly scheduled meeting, is of value.

An Office Environment

Since I picked it up in grad school, I've been fascinated by a book called Peopleware and the different ways of thinking about the work place.

This morning I read a bit about how a work place or work space affects the productivity of a software developer:

"Staying late or arriving early or staying home to work in peace is a damning indictment of the office environment. The amazing thing is not that it's so often impossible to work in the workplace; the amazing thing is that everyone knows it and nobody ever does anything about it."
- Chapter 8, "You Never Get Anything Done Around Here from 9 to 5"

They went on to describe a study they did involving developer productivity. They found the normal 10:1 range of individual developer productivity. What was surprising was they also found that there was a 10:1 or so range for organizational productivity - with two developers from each organization. The two developers from each performed on about the same level.

It would seem that not only individual developer productivity matters, but also their working environment.

Using firefox extensions in mozilla prism

Matt Gertner, creator of the Mozilla Prism firefox addon/SSB, just posted some basics on how to get a firefox extension working with prism.

Thought it was interesting for those messing with site-specific browsers...

http://browsing.justdiscourse.com/2009/10/22/prism-and-extensions/

A Pidgin plugin for Growl for Windows

I've been fond of Growl on the mac as a pretty standard notification mechanism.

Growl for Windows is a project that uses the same protocols to do notification on the windows side.

After a discussion in the forums, someone has finally implemented support for one of the last core apps that was missing for a long time: pidgin.

http://blog.growlforwindows.com/2009/10/pigdin-growl-sittin-in-tree.html

Previously there was a snarl->growl bridge called gnarly that would notify growl of messages, but it's nice to see that such a popular app is getting some attention :).

W00t for open source :).

FindBugs in IntelliJ IDEA

I was just perusing the plugin repository for IntelliJ IDEA plugins recently and came across the FindBugs plugin. It will analyze your code and give you a categorized list of potential problems in your code that links to the source. It bundles the latest FindBugs implementation so there is no need to download that as well.

It looks promising for catching things I may have missed in my code.

There is currently a bug where when you first use it, you need to go into the settings for FindBugs and click on Restore Defaults. That initializes the list of detectors that it uses.

Links:
http://findbugs.sourceforge.net - the findbugs home page
https://findbugs-idea.dev.java.net - the findbugs IntelliJ IDEA home page
http://www.jetbrains.com/idea - the IntelliJ IDEA home page

Additional stuff from Developing with Mozilla Prism

Yesterday in my presentation on Developing with Mozilla Prism, I had some difficulties getting an app to work because of an intermittent bug in prism right now.

I did want to post the webapp.js code for the Google Reader prism app that I was showing during the presentation:

/**
* Google Reader web app script
*/

function n(app, msg) {
 window.platform.showNotification(app, msg, null);
}

/**
* standard webrunner plugin api
*/
function startup() {
}

function preload() {
}

function load() {      
   Reader.load();
}

function shutdown() {
   Reader.shutdown();
}

function error() {
}

var Reader = {
   _timer : null,
   _unreadCount : 0,
   _window : null,

   receiveMessage : function() {    
       //window.platform.sound().beep();
       window.platform.getAttention();

       n("Google Reader", "You have " + Reader._unreadCount + " unread item(s).");
   },

   _init : function() {
       Reader._unreadCount = 0;
       Reader._window = window;
   },
 
   run : function() {
       if (Reader._unreadCount == undefined)
       Reader._unreadCount = 0;

       var title = Reader._window.top.document.title;

       var matches = title.match(/Google Reader \((\d+)\)/);
       if (matches) {    
           if (matches[1] > Reader._unreadCount) {
                Reader._unreadCount = matches[1];
                Reader.receiveMessage();
           }

           Reader._unreadCount = matches[1];
       }
   },

   load : function() {  
       Reader._init();
       // kick off a polling timer to check for new articles
       Reader._timer = Reader._window.setInterval(Reader.run, 5000);
   },

   shutdown : function() {  
       if (Reader._timer)
           Reader._window.clearInterval(Reader._timer);
   }
};

Also, the MIME type to add to your httpd.conf to allow prism webapp bundles to load properly by linking to them is:

AddType application/x-webapp .webapp

Talking with Matt Gertner a little about adding javascript to your own web page... He said that you could use the window.platform stuff in your webapp and that would, in many cases, simply replace the need for webapp.js since your web page would then essentially become prism aware. That's another option for trying to distribute a prism app.

For more information on prism - check out https://developer.mozilla.org/en/Prism

Slides for Developing with Mozilla Prism

I just wanted to post my slides for my presentation this afternoon on Developing with Mozilla Prism.

It will be in the Auditorium at 3 PM. I hope to see you there!

I received a bag of stuff from Mozilla to give away - various fun mozilla/firefox stickers and mozilla firefox badge lanyards.

Slides (pdf through Google Docs)

Utah Open Source conference 2009 - Developing with Mozilla Prism

I will be speaking at the Utah Open Source conference next Thursday at 3 PM on Developing with Mozilla Prism.

I'm excited because Mozilla Prism is nearing a final release of 1.0 - it's at 1.0 b2 at this point. I talked to a Mozilla employee, Mark Finkle, who works on Prism, in addition to Fennec, their mobile browser project. He talked to Mozilla HQ and it sounds like I'll be able to get some Mozilla swag to give away at the presentation!

I'm also trying to start a community around building a prism webapp bundle library that I'll talk about more in my presentation. I thought it would be nice to have some open source examples of Prism web application bundles that people could refer to and add to. So I created a Google Code project called prism-apps.

In any case, check out the Prism project and come to the presentation!

(I'll post my slides later in the week as I finalize them.)

Subtle difference - extending ListResourceBundle vs ResourceBundle

I've been mucking about extending ListResourceBundle and then just plain ResourceBundle trying to get things to work with a base case - a custom resource bundle.

Taking a look at the Java i18n tutorial, extending ListResourceBundle doesn't require subclassing the default bundle; see here.

Then taking a look at the ResourceBundle javadocs and their custom resource bundle example, you *do* have to extend the default bundle; see here.

So:

public class MyListResources_it extends ListResourceBundle {
  ...
}

as opposed to:

public class MyResources_it extends MyResources {
  ...
}

Kind of a confusing inconsistency - maybe the ListResourceBundle is just smarter about its look up.

Internationalization - dynamic and in a database

So I've been tasked with doing some internationalization for a web application in a less conventional way. I thought it would be useful to put together some thoughts as I get started on what I think will be a decent solution.

From what I understood, internationalization in Java (the language I'm using) involves properties files and locales and such. In the project I'm on, there is a requirement to be able to update the translated texts on the fly, without a server restart. This complicates the using of properties files a bit, since writing to those files seems a bit hackish and then what do I do to reload them.

So we've decided to look into a database backing for the whole business since that's pretty dynamic and accessible to the people - either via some database interface or an admin tool - working on the text.

We were kicking around various options to do something custom, a custom map or something and redirect whatever framework we were using.

Then I found some methods native to Java - no framework required - which should handle it nicely.

1. ListResourceBundle has been around for a while in Java-land, but it's a nice way of not using properties files, but classes as bundles for different locales. This seems like a great way to get those database backed properties into the mix.
2. Java 6 also has some nice customization options for how resource bundles get loaded. Specifically, ResourceBundle.Control has some options to customize the time-to-live of the cache for the resource bundle, from no-cache at all, to specifying a timeframe in milliseconds for how long a cache is valid.

So going forward, it looks like I'll need to mess with both for making something that's as close to the wire as I can - which allows me to not have to re-code caching and other things specific to a system for i18n.

Having intellij + tomcat update web content without a restart

So I've had issues with intellij + tomcat updating web content without a tomcat restart - very painful.

I've found the magic that will allow for it thanks to a coworker.

There are three things to do:

1. Make sure your webapp is an exploded directory instead of a war (not sure if this is required)
2. Set Settings->Debugger->HotSwap->Reload classes after compilation to Always - if it can't do it, I'll have to restart anyway
3. Set Settings->Compiler->Deploy web applications to server after compilation to Never

Then after updating a page, on Windows press ctrl-shift-F9 to "compile" the page or re-put it into the exploded directory.

My sticking point was number 3 - it would put the updated file out there, but when it deployed the web application, tomcat got all confused and required a restart - I think probably because everything was updated including the web.xml.