Twitter Updates

Tutorial days from OSCON 2010

July 25th, 2010

I had fun at OSCON last year, so I decided to go back this year. It’s incredible experience being part of hundreds of developers (more than 2500), who are excited about open source and upcoming technogolies. Though, there were a number of tracks, but I was mainly interested in Mobile Computing, Cloud Computing, No-SQL, and Scala. There were also interesting sessions on Emerging Languages and Hardware Hacking with Arduino, but I didn’t get chance to attend them.

Tutorials Day One

I arrived in Portalnd on Sunday night and the first session I attended on Monday morning was on Android. I saw fair number of Android phones at the conference and people were excited about Android. There was a lot of bashing of Apple, but I won’t get into it. Here are some of the highlights from the Android session:

Android for Java Developers

Android is a specially designed O/S based on Linux that uses a number of open source libraries and custom framework for building mobile applications. It even comes with a command line tool adb that opens shell directly on the device or emulator. Though, Android applications are written in Java but the byte-codes are converted into Dalvik instruction sets (.dex files). Dalvik is a register based Java VM as opposed to Sun’s stack based and uses JavaSE minus Swing/AWT APIs. You use Dalvik generated executable and resources (images, audio, configurations, etc.) to build the application file (APK), which is then signed by self-generated certificate. The best part of Android development is their deployment, which is an order of magnitude easier compare to Apple iOS.

Android SDK

Before developing, you would need to download the Eclipse IDE, Android SDK and then install Eclipse plugin from https://dl-ssl.google.com/android/eclipse/.

Creating Android Device

One of the downside of Android is variations of the capabilities of all Android devices, however you don’t need to own all variations. All you need is to create Android device by giving name, target version, SD card, skin and other hardware limitations. Android comes with emulator as opposed to simulator with iPhone SDK and follows hardware much more closely. On the downside, it takes a long time to start the Android emulator, but on the upside you can emulate call, SMS, and launch multiple emulators, which can other emulators. Note, the android device file definition goes into ~/.android/avd directory.

Hello World

You can create a new project and specify the build target, application name, package name, and main activity class. An activity class represents a controller class, which you define for each screen. Android uses UI to generate layout file (R file). If your application need resources such as images and property files, you store them in res directory. You can also create resource files for storing strings to make your application easily localized for various languages.

Activities

Activities represent screens and are managed by the activity managers. The activity can be in one of five states: starting, running, paused, stopped, and destroyed. The Activity class defines callback methods such as onCreate, onStart, onRestoreInstanceState, and onResume when the state changes.

Intent

Intent represents events or actions, which can be explicit or implicit.

Services

One of the key advantage of Android has been its support of multi-tasking and you can have a background processing using services. Services also have lifecycle, but simpler than activities.

Content Providers

Content providers allow sharing data between applications such as contacts, mediastore, settings, etc.

Broadcast Receivers

These allow pub-sub based mechanism for system events such as SMS messages.

Architecture of Android applications

Here are some of the tips that Marko Gargenta gave when designing application:

  • Isolate I/O operations such as network or disk operations into separate tasks or background services, which either use notification or database to communicate with interactive application. For example, in his sample twitter application, he used background service to poll tweets and stored them to the database. The activities then polled tweets from the database and also subscribed to the notification when new tweet arrives.
  • Use layout for screen design as it is more declarative and separate text values from the layout and use string resources.
  • Due to the variations of the Android devices, use layout weight and density intensity pixel (dp or sp) instead of fixed pixels (px) for components.
  • Android SDK provides GUI tool for designing layout and you will need to bind the UI components back to the activity classes, so use consistent naming convention for both layout file and the activity file, e.g. 
       public class Twitter extends Activity {
     EditText editTweet;
     Button buttonUpdate;
     public void onCreate(...) {
       editTweet = (EditText) findViewById(R.id.editTweet);
       editButton = (Button) findViewById(R.id.editButton);
       buttonUpdate.setOnClickListener(this)
         String tweet = editTweet.getText().toString();
       }
     }
   }

Adapters

Android allows easily access large datasets as arrays that can be displayed on the screen.

Logging

Android uses custom libc and Java for Logging and you can add logging as:

   Log.debug("ClassName", "button clicked");

You can use “adb logcat” command to view logs, e.g. “adb logcat Twitter:* *:S”

Security Permissions

Any permissions that user need must be defined in the manifest file.

Advanced components

Android comes with a number of advanced components such as Map, Menus, Graphics, Animations, Multimedia, Preferences, Sqlite databases, etc.

Cloud to Device Push

This is a new service similar to iPhone push notification.

Debugging

Marko suggested use of Eclipse debugger, logcat, hierarchy viewer and traceview for debugging.

For more information on Android session, Download the slides.

The Seductions of Scala

For the second half of the day, I attended Dean Wampler’s session on Scala. I have been long interested in Scala and have done a little development on my own. As OSCON offered a lot of sessions on Scala, I took the opportunity to learn more on Scala. Dean highlighted concurrency, concise code, and correctness, better object model as major benefits of Scala.

Introduction

Scala comes with a number of features for concise code such as implicit type conversion, properties with uniform access principle and optional semicolons and paranthesis when function arguments are one or empty. Scala allows objects to act as function using apply method, e.g.

   class Logger(val level: Level) {
         def apply(message: String) = {
                 log(level, message)
         }
   }

 val error = new Logger(ERROR) ...
 error("Network error.")

Also, Scala treats primitive types as objects, but are comiled down as primitivies. Scala also treats functions as objects, e.g. you can create list or map without new

 val list = List(1, 2, 3, 4, 5)
 val map = Map("name" -> "Dean", "age" -> 39)

Above list syntax is same as 1::2::3::4::5::Nil. Scala also allows any symbol for function name so you can define functions that look like operator overloading. Scala uses infix operator notation, for example, following two expressions are equivalent:

 "hello" + "world"
 "hello".+("world")

Scala gives you higher level operations such as map, filter, fold/reduce, e.g.

 val list = "a" :: "b" :: Nil
 list map {
         s => s.toUpperCase
 }

Generics

Martin Ordesky added limited support of Generics in Java, but he added fully functional generics support in Scala, e.g.

 class List[A] { ...
 def map[B](f: A => B): List[B]
 ...
 }

Traits

One of nice feature of Scala is its support of Traits, which are interfaces with implementation and are similar to Ruby mixins. Here is an example:

 trait Logger { def log(level: Level,
 message: String) = { Log.log(level, message)
 } }
 val dean = new Person(...) extends Logger
 dean.log(ERROR, "Bozo alert!!")

Scala also defines traits for functions to convert them into objects, e.g.

 trait Function1[A,R] extends AnyRef {
 def apply(a:A): R
 ... }

User-defined factory methods

You can define functions as factory-methods to instantiate objects, e.g.

  val persons = Map("dean" -> deanPerson, "alex", -> alexPerson)

DSL

Scala offers powerful semantics to define internal DSLs, e.g. you can create your own controls, e.g.

 import java.io._ object Loop {
 {...}
 }
 def loop(file: File, f: (Int,String) => Unit) =
 ...
 loop (new File("...")) { (n, line) => ...
 }

Options as alternative to Null

Scala avoid NullPointerExceptions by wrapping nulls into options, e.g.

 abstract class Option[T] {...} case class Some[T](t: T)
 extends Option[T] {...} case object None
 extends Option[Nothing] {...}

Case Classes

Case classes provide succint syntax for creatng Javabeans.

For comprehensions

Scala provides for comprehensions, which are similar to Python generators, e.g.

 val l = List( Some("a"), None, Some("b"), None, Some("c"))
 for (Some(s) <- l) yield s

Actors

Scala provides Actor based concurrency similar to Erlang, though there are multiple implementations and Akka seems to provide better implementation than what comes with Scala. Here is an example:

 case class Point(
 x: Double, y: Double)
 abstract class Shape { def draw()
 }
 ....
 package shapes import scala.actors._, Actor._ object ShapeDrawingActor
 extends Actor { def act() {
 loop { receive {
   case s:Shape =>
   s.draw()
 ... }
 } }
 }

Tutorials Day Two

On day two, I attended all-day Scala summit, which covered various topics for practical Scala.

Why Scala?

The summit started with session on "Why Scala?" by Alex Payne and Dean Wampler. Dean repeated some of same concepts from Monday's session on Scala's conciseness, concurrency, correctness, infix operator, type inference, case classes, etc. Dean then gave some examples of actors using Akka, where he calls multiple services using actors and then gather the results, e.g.

 val futures = for { s   <- services
 server <- allServersFor(s) }
 yield (server !!! HeartBeat)
 Futures.awaitAll(futures) val results = for {
 future <- futures
 result <- future.result } yield result val all = results reduceLeft(
 (r1, r2) => r1 merge r2 ) compact(render(all))
 }}

Akka: Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Actors

Jonas Bonér then gave brief overview of Akka, which provides a number ofabstractions for concurrency such as actors, STM, and agents. Jonas gave introduction to actors, which provide concurrency based on message-passing, shared-nothing, and mailbox. In Akka, actors can be thread based or event based, where event based actors are very light-weight and you can create millions of them (each actor takes about 600 bytes as opposed to 300 bytes in Erlang).

factory-methods

Akka uses factory-methods to hide type of actors, e.g.

 val counter = actorOf[Counter]  // counter is ActorRef
 actor.start
 actor.stop

Jonas suggested use of actorOf as opposed to "new Counter" syntax as it avoids calling methods on the objects directly. Akka uses !, !! and !!! notations to send messages, where ! is just fire and forget, !! collects results using Futures and !!! returns Future, e..g

 counter ! Tick // send message -- fire-forget
 val result  = (actor !! Message).as[String] // uses Future under the hood with timeout
 val resultOption = actor !! Message
 val result = resultOption.getOrElse(defaultResult)
 val result = resultOption.getOrElse(throw new Exception("time out"))
 val future = actor !!! Message
 future.await
 val result = future.get

 Futures.awaitOne(List(fut1, ..))
 Futures.awaitAll(List(fut1,))

You use self.reply to reply back to the sender and access sender using self.sender or self.senderFuture.

Immutable Messages

In order to keep actors free of side effects, messages must be immutable using case classes, tuples or lists, e.g.

 - case class Register(user: User)
 - actor ! Register(user)
 - actor ! (username, password)
 - actor ! List("xxx", "yy")

Dispatchers

Akka comes with a number of dispatches such as event based, thread based, reactor, etc. See Dispatchers class for more information.

Queues

Akka also comes with various queue types such as unbounded LinkedBlockingQueue, bounded LinkedBlockingQueue, etc.

ActorRegistry

ActorRegistry provides lookup methods for actors such as ActorRegistry.actorsFor.

Fault tolerance

Akka borrows concepts of hierarchy of supervisors for managing actors or processes from Erlang. Erlang's philosophy for fault tolerance is let it crash and the supervisor automatically starts failed process or group of processes. You use link(actor), unlink(actor), startLink(actor) to connect actors with supervisors and trap events using trapExit = List(classOf[ServiceException], classOf[PersistentException]), e.g.

 class Supervisor extends Actor {
   import self._
   trapExit = List(classOf[Throwable])
 }

 class FaultTolerantService extends Actor
   override def preRestart
   override def postRestart

Remote Actors

You can start a node, which is remotely accessible using:

 RemoteNode.start("localhost", 9999)
 spawnLinkRemote[MyActor]("darkstar", 9999)

 RemoteNode.register("service:id", )

STM, Transactors, Modules, Camel, Storage

We ran out of time for the rest of contents, but you can read more from the Slides.

Simple Build Tool

Next, Mark Harrah presented SBT, which everyone raved at the summit. SBT uses Scala based DSL for writing build scripts and internally uses Ivy for managing dependencies. You can create a new project by creating a new directory and typing sbt. You can set various properties in sbt shell such as target version, e.g.

 set build.scala.versions 2.80
 reload

You can easily create custom tasks in sbt by extending DefaultProject, e.g.

 import sbt._
 class MyProject(info: ProjectInfo) extends DefaultProject(info) {
   lazy val hi = task { println("Hi"); None}
   lazy val goodbye = task { println("Bye"); None} dependsOn(hi)
 }

You can also target tasks for just test using

 import sbt._
 class MyProject(info: ProjectInfo) extends DefaultProject(info) {
   val sc = "org.scala-tools.testing" %% "scalacheck" % "1.7" % "test"
 }
 OR
 import sbt._
 class MyProject(info: ProjectInfo) extends DefaultProject(info) {
   val sc = "org.scala-tools.sbt" %% "launcher-interface" % "0.74"
   val tt = "org.scala-tools.sbt" %% "launcher-interface" % "0.74" % "test"
 }

You can define main application as follows:

 import xsbti._
 class HW extends AppMain {
   def run(config: AppConfiguration): MainResult = {config.arguments foreach println; new Exit {def code = 0}}
 }

You generate executable jar by typing publish-local in sbt shell. You can define plugins as follows:

 class Plugins(inf: ProjectInfo) extends PluginDefinition(info) {
 val android = "org.scala-tools.sbt" % "sbt-android-plugin" % "0.5.0"
 }

Finally, sbt allows you to create processors, which behave like scaffolding in Rails, e.g.

 import xsbti._
 import processor._
 class HW extends BasicProcessor {
   def apply(project: Project, args: String) {
     import project._
         val contents = "This is " + name + "" + version + "\n" + args + "\n"
         FileUtilities.write(info.projectPath / "README" asFile, contents, log)
   }
 }

When you type publish, it will create README file for the project. That was pretty much the introduction to the sbt.

Specs & Scala, Tips and Tricks for a Friendly DSL Syntax

Eric Torreborre talked about Spec, which a BDD based testing tool for Scala. Spec provides support for BDD, Structures, Matchers, ScalaCheck, Mocks, Runners and databases. You use matchers to compare strings or XML contents, e.g.

 class Reverse2Spec extends  Specficiation {
   reverse("") must_== ""
 ...

You can restrict scope by defining tag method, e.g.

 class Example(des : String) {
   def in(arg: Any) = expectations
   def tag(t : String) = this
 }

Scala DSL

Spec uses a number of tricks for simplifying the syntax such as implicit parameters, operators, lazy evaluation. 

 "With a 3 tests ok" in {
   prop
 }
 //Some paraemeters can be implicit
 implicit val defaultparams = new Params

 "this should not explode" in {
   error("boom")
 }
 def in(e: => Any) =                                     // parameters are evaluated lazily when you use it

It also uses principles such as add principle by adding new functionality, e.g.

 result + 1
 result.pp + 1

Spec also supports table similar to Fit and Fitness for writing concise tests. Overall, I was impressed with wide set of tools for writing tests.

Lift: Quick and Fun

Lift is a Scala based web framework for writing secure, typesafe, concise, and interactive (like desktop) applications. It abstracts much of plumbing of HTTP, which I personally don't like as I have found web frameworks that does that results in leaky abstractions. Lift also uses stateful web applications, which require sticky sessions, which is another area that I have found to be problematic for scalability and upgrade. Here is an example of Lift chat server:

 package code.comet
 import net.liftweb._
 import http._
 import actor._
 import scala.xml.
 object ChatServer extends LiftActor withListenerManager {
         private var msgs = List("Welcome")
         def createUpdate = msgs
         override def lowPriority = {
                 case s: String => msgs ::= s; updateListeners()
         }
 }

 class Chat extends CometActor withCometListener {
   private var msgs: List[String] = Nil
   def regiserWith = ChatServer
   override def lowPriority = {
      case l: List[String] = msgs = l; reRender(false)  // don't use reRender
   }
   def line(in: NodeSeq) : NodeSeq = msgs.reverse.flatMap(m => bind("chat", in, "item" -> m))
   def render = bind("chat", "line" -> line _)
 }

In Lift, every component has GUID and version that was used to render and then sets up long poll and then receive deltas (every 100ms). You can use sbt to deploy jetty and prepare war file, e.g.

 sbt
 >jetty-run
 >prepareWeb                                                     // uses JRebel to reload classes

Rewiring Android with Scala

This was another interesting talk by Nathan Hamblen for using Scala for writing Android applications. The key selling point of Scala has been conciseness of the language, and you can write simple code such as:

 dialog.setOnShowListener { di: DialogInterface =>
   runSomeCode()
 }

instead of 

 dialog.setOnShowListener(
   new DialogInterface.OnShowListener() {
     public void onShow(DialogInterface interface) {
       runSomeCode();
     }
   }
 );

or 

 future { runSomeCode(myObject) }

instead of

 new AsyncTask () {
   protected Integer doInBackground(MyObject... objs) {
     runSomeCode(objs[0]);
   }
 }.execute(myObject);

Nathan showed how you can define Scala traits to add lazy handlers for android code, e.g.

 trait ScalaActivity extends Activity {
 ...
 lazy val handler = new Handler
 def post(block: => Unit) {
   handler.post(new Runnable{
     def run { block }
   })
 }

Or you can extend APIs, e.g.

 implicit def f2cancel(block: DialogInterface => Unit) =
   new DialogInterface.OnCancelListener {
     def onCancel(dialog: DialogInterface) {
       block(dialog)
     }
   }
 ...
 new AlertDialog.Builder(this)
   .setOnCancelListener {
     di: DialogInterface => finish()
   }

Nathan also showed a plugin (sbt-android-plugin) to create type-safe layout instead of using R.java file generated by Android, which you can get it from git clone git://github.com/meetup/meetabout.git. On the downside, Scala based android applications require Scala jar files and the size of application becomes considerable large. Though, you can use tools to extract the classes that you need, but it would still be larger than Java code.

Scala in Practice

Alex Payne and Coda Hale had a section on Scala in practice, but it was only Q/A session. I was a bit disappointed that they didn't come prepare with actual usage or war stories from their work environment.

High Wizardry in the Land of Scala

The last section of the day was a bit on type and category theory, which was interesting but mostly theortical. Daniel Spiewak explained difference between kind and type system. The only tip from the session I got was that Values are to types as types are to kinds. Finally, Daniel explained that newly released 2.8.0 version of Scala supports continuation but it's all broken and useless.

Summary

Overall, I found sessions on both Android and Scala were well worth the time and it peaked my interest in both. I think the ecosystem of Scala has matured and there is better tools support with the new version (2.8). I am going to try to influence co-workers into using it for new development. I am also going to start Android project pretty soon but I am a bit hesitant on writing in Scala due to increased application size.

Posted by admin
Filed in Computing
No Comments »

A few lessons from Seth Godin’s book – Linchpin: Are You Indispensable?

June 3rd, 2010

I just finished reading Seth Godin’s new book Linchpin: Are You Indispensable?. Seth shows how the white-collar jobs, which supposed to save the middle class are being eliminated either by machines or outsourcing with cheap labors. He shows that you can either continue to live your life as a faceless cog or choose to become Linchpin. Here are some of the lessons I learned from this book:

Industrial Revolution is Over



The race to make average stuff for average people in huge quantities is almost over.

This book shows the industrial revolution is changing and in order to survive in the new era of economy, you have to become linchpin or indispensable. In last three hundred years, the industrialization began by standardizing the tasks so that it can be performed by easily replaceable labor or so called cogs. It relied on two layers: management and labor, where management breaks production of goods into tiny tasks, which are performed by the labor. The management wins when it can get the most work for the least pay. The system taught workers to follow the instructions and you don’t have to think. Though, that system worked but has been falling apart in the face of competition, outsourcing and globalization. The attendance-based compensation (ABC) is over. Th old American dream that taught to keep your head down, follow instructions, work hard and you will be rewarded is dead. The mass production treats everything such as labor and material as interchangeable. However, in global market, the competition is fierce and cheap strategy doesn’t scale very well.
Instead of easily replaced laborers or cogs, you can choose to become Linchpin by differentiating yourself from the rest and focusing on humanity, connection and art. The web has made it easier to be productive and create or invent. The new American dream is to be remarkable, generous, create art and connect with people.

Education System is a Sham



In capitalist market, the companies make money by hiring obedient and competent workers as cheaply as you can and using productivity advantage to earn more profit. Andrew Carnegie saw that limited amount of education to get them to cooperate. The school system throughout the world encourages mediocre obedience and is driven by fear as when we learn things in fear. Seth shows public school system is designed to prepare us for factories, where we are just replaceable cogs and care little about our jobs or customers. The same factory model created consumer culture that uses consumption as a shortcut to happniess. Instead, school should teach solving interesting problems and leading.

Becoming a Linchpin



In order to become a linchpin or indispensable, you must embrace an artist and genius within you. Seth recommends avoiding asympototic goals such as bowling, where there is a ceiling of how good you can be. Also, for an artist, the economy is not just zero sum game, instead he/she can increase the pie. Seth cites Richard Florida’s survey of top ten reason for employees to do best work as follows:

  • challenge and responsibility
  • flexibility
  • stable work environment
  • money
  • professional development
  • peer recognition
  • stimulatng colleagues and bosses
  • exciting job content
  • organization culture
  • location and community

All of above reasons except money are internal that we can control. Seth encourages readers to find the work that suits your passion. He uses Emotional labor term, originally coined by Arlie Hochschild to connect with the work. Though, you may get a little compensation in return of emotional labor, but you get inward reward. Instead of day’s work for day’s job or the poverty mentality that treats life as zero sum game, you give gift and build bonds. Seth shows that the easier work is to quantify, the less it’s worth and more humanity you bring to your work, the better results you will receive. Seth cites Krulak’s law for building strong relations with your customers, i.e.,

The closer you get to the front, the more power you have over the brand.

Resistance to Change



Seth gives plenty of examples and demonstrates that real artists ship, however shipping is hard due to trashing/tweaking and coordination. According to Seth, the biggest resistance to the change is our lizard brain. He explains how we all have two brains: primeval brain or lizard brain and gray matter or recently developed brain. The lizard brain has animal instincts such as hungry, scared, angry and horny, whereas newer brain allows big thoughts, generosity, speech, and art. Lizard brain seek compfort and obedience, and avoids risks, public speaking and generosity.

Good is enemy of perfect



Seth encourages readers to become excellent and not perfect as art is never defect-free. He cites Bre Pettis, who says that there are three states of being: not knowing, action and completion. He says accept that everything is draft as it helps to get it done.

Generosity



Exchanging gifts is an ancient tradition. Seth shows that artists who give gifts win as becoming a linchpin is not an act of selfishness. Seth also shows how usury was prohibited in Bible as interest-free loan was kind of gift. This changed when Martin Luther lifted the sanction to get support for the Protestant Reformation. Seth writes:

For the last five hundred years, the best way to succeed has been to treat everyone as a stranger you could do business with.

Seth cites Metcalfe’s law states that the value of a network increases with the square of the number of nodes on the network. The new social media platforms such as Twitter, Facebook, Blogsphere, and Internet is changing the circle of the gift system and he shows that there are three cicles of gifts, the first circle represents true gifts to family and friends. The second circle is for commerce, they pay for souvenir edition and the third circle is your tribe, followers, fans or friendlies.

There is no map



Seeing the future is hard because we are attached to the world and want stability and fear change. Seth gives plenty of examples of record industry and newspaper industry who have been too attached with their legacy model and failed to adjust in the new economy. In order to become linchpin, you need to draw a map and lead instead of being passive. You need to find a job that matches your passion.

Culture of connections



The industrialization removed human connection between different parties. The social media and Internet is changing that, now companies can connect directly with their customers and receive their input. Often, when companies negotiate with other companies, the key point of distinction is the perceived connection between the prospect and the organiziation. The salesman who relies only on the script would fail, instead you have to rely on honest signals and genuine gifts to make connections.

Seven attributes of Linchpin

Linchpins are geniuses, artists and givers of gifts, who extert emotional labor and make their own map. Here are seven abilities of the linchin:

  • Providing a unique interface between members of the organization
  • Delivering unique creativity
  • Managing a situation or organization of great complexity
  • Leading customers
  • Inspiring staff
  • Providing deep domain knowledge
  • Possessing a unique talent

Conclusion

We have been in declining economy for a while and many of the white collar and blue collar jobs lost in last few years won’t come back. I blogged a few years ago on Offshoring and wrote:

Blue collar jobs have been moved to offshore with a little uproar, and offshoring of white collar jobs in IT have received not much fret publically either. Clearly, in all this people are losers and multinational
companies are winners.

I found a lot of Seth’s advice similar to agile movement in software development and My Job Went to India. I also wrote about Taylorism in my blog IT Sweatshops, where I deplored Taylorism based command and control structure in a lot of companies even the one that claim to adopt agile methodologies. Seth even says that you don’t need a resume as it hides the fact that you are linchpin. Instead have a project that an employer can see or blog that people can follow. I find this book offers very practical and timely advice for future market. In the job market, You need to differentiate yourself and have a trail of breadcrumbs of your previous work. Being an average is over, instead you have to be a linchpin and live without a map.


Posted by admin
Filed in Business
No Comments »

Validating receipts from Apple iPhone store in Ruby

May 24th, 2010

Recently, I had to write a service on Ruby-on-Rails project to validate an in-store purchase for iPhone application. Though, it was fairly straight forward, but I am posting the code in case someone else needs it:

  1 require 'net/http'                                                                                                                                                                                                                                                            

  2 require 'net/https'
  3 require 'uri'
  4
  5 class AppleReceiptVerifier 

  6   #
  7   ### Verifies Apple receipt submitted by iPhone 
  8   ### See http://developer.apple.com/iphone/library/documentation/NetworkingInternet/Conceptual/StoreKitGuide/Overview%20of%20the%20Store%20Kit%20API/OverviewoftheStoreKitAPI.html#//apple_ref/doc/uid/TP40008267-CH100-SW14

  9   #
 10   def self.verify(b64_receipt)
 11     url = URI.parse(APPLE_RECEIPT_VERIFY_URL)

 12     http = Net::HTTP.new(url.host, url.port)
 13     http.use_ssl = true
 14     http.verify_mode = OpenSSL::SSL::VERIFY_NONE

 15     valid = false
 16     json_request = {'receipt-data' => b64_receipt}.to_json
 17     resp, resp_body = http.post(url.path, json_request.to_s, {'Content-Type' => 'application/x-www-form-urlencoded'})

 18     if resp.code == '200'
 19       json_resp = JSON.parse(resp_body)
 20       if json_resp['status'] == 0

 21         valid = true
 22       end
 23     end
 24     valid
 25   end 

 26 end
 27
 28

Note that the receipt returned by iPhone APIs is not base 64 encoded so you will need to encode it before calling the service. Also, for testing, APPLE_RECEIPT_VERIFY_URL will point to the sandbox environment, i.e., https://sandbox.itunes.apple.com/verifyReceipt and for real purchase it will point to https://buy.itunes.apple.com/verifyReceipt. Finally, you can learn more from the Apple documentation on purchase model and on validating receipts.


Posted by admin
Filed in iPhone development
No Comments »

Building a stock quote server in Erlang using Ejabberd, XMPP, Bosh, Exmpp, Strophe and Yaws

May 10th, 2010

Recently, I have been building a stock quote server at work that publishes financial data using using Ejabberd, XMPP, PubSub, Exmpp and Bosh on the server side and Strophe library on the web application front. I will describe a simplified implementation of the quote server using Yahoo Quotes.

Installation

Download Ejabberd and go through the installation wizad. You will be asked your host name, admin account/password and whether ejabberd would be running in a clustered environment. For this tutorial, we will be running ejabberd on a single. Once installed, you can start the ejabbered server using

 /Applications/ejabberd-2.1.3/bin/ejabberdctl start

As, I am using Mac, the actual path on your machine may be different. The ejabbered comes with a web baesd admin tool, that you can access using

 http://<your-host-name>:5280/admin

and you would be able to see available nodes, users, etc.




Registering Users

We will be creating two users: producer and consumer, where the former would be used for publishing stock quotes and latter would be used for subscribing quotes on the web side, i.e.,

 sudo /Applications/ejabberd-2.1.3/bin/ejabberdctl register producer  producer
 sudo /Applications/ejabberd-2.1.3/bin/ejabberdctl register consumer  consumer

Debuging with Psi

You can debug XMPP communications using a jabber client such as Psi, which you can download. After you download, you can install and specify your local hostname as a server, e.g.





You can then login using consumer@<your-host-name> with password consumer. As, we will be using PubSub protocol, you can discover available nodes or topics using General->Service Discovery from the menu, e.g.




Downloading Sample Code

I have stored all code needed for this example on http://github.com/bhatti/FQPubSub, that you can checkout using:

 git clone git@github.com:bhatti/FQPubSub.git

The sample code depends on exmpp, lhttpc, jsonerl, and yaws modules so after downloading the code, checkout dependent modules using

 git submodule init
 git submodule update

Above commands will checkout dependent modules in deps directory.

Building Sample Code

Before building, ensure you have make and autoconf tools installed, then replace <paraclete.local> with your <your-host-name> in docroot/index.html and src/quote_utils.hrl. Then type following command

 make

to build all sample code and dependent libraries

Starting Web Server

Though, the web code including Srophe library and Javascript can be run directly in the browser, but you can start Yaws to serve the application as follows:

 erl -pa ebin deps/exmpp/ebin/ deps/lhttpc/ebin/ deps/yaws/ebin -boot start_sasl -run web_server start

Note, that the web server will be continuously running, so you can open a separate shell before typing above command.

Publishing Quotes

Create two separate shells and type following command in first shell:

   erl -pa ebin deps/exmpp/ebin/ deps/lhttpc/ebin/ deps/yaws/ebin -boot start_sasl -run quote_publisher start AAPL

and following command in second shell

   erl -pa ebin deps/exmpp/ebin/ deps/lhttpc/ebin/ deps/yaws/ebin -boot start_sasl -run quote_publisher start IBM

Above commands will start Erlang processes, that will poll Yahoo Quotes every second and publish the quotes on the node AAPL and IBM respectively.

Next point your browser to http://<your-host-name>:8000/, and add “IBM” and “AAPL” symbols, you would then see quotes for both symbols, e.g.

Code under the hood

Now that you are able to run the example, let’s take a look at the code how it works:

Client library for Yahoo Finance

Though, at work we use our own real time stock quote feed, but for this sample I implemented stock quote feed using Yahoo Finance. The src/yquote_client.hrl and src/yquote_client.erl define client API for accessing Yahoo finance service. Here is the Erlang code for requesting the quote using HTTP request and parsing it:

  1 %%%-------------------------------------------------------------------

  2 %%% File : yquote_client.erl
  3 %%% Author : Shahzad Bhatti
  4 %%% Purpose : Wrapper Library for Yahoo Stock Quotes

  5 %%% Created : May 8, 2010
  6 %%%-------------------------------------------------------------------
  7
  8 -module(yquote_client).

  9
 10 -author('bhatti@plexobject.com').
 11
 12 -export([
 13          quote/1
 14         ]).

 15
 16 -record(quote, {
 17         symbol,
 18         price,
 19         change,
 20         volume,

 21         avg_daily_volume,
 22         stock_exchange,
 23         market_cap,
 24         book_value,
 25         ebitda,
 26         dividend_per_share,

 27         dividend_yield,
 28         earnings_per_share,
 29         week_52_high,
 30         week_52_low,
 31         day_50_moving_avg,
 32         day_200_moving_avg,

 33         price_earnings_ratio,
 34         price_earnings_growth_ratio,
 35         price_sales_ratio,
 36         price_book_ratio,
 37         short_ratio}).
 38 

 39
 40
 41 quote(Symbol) ->
 42     inets:start(),
 43     {ok,{_Status, _Headers, Response}} = http:request(get, {url(Symbol), []},

 44         [{timeout, 5000}], [{sync, true}]),
 45
 46     Values = re:split(Response, "[,\r\n]"),
 47     #quote{

 48         symbol = list_to_binary(Symbol),
 49         price = to_float(lists:nth(1, Values)),
 50         change = to_float(lists:nth(2, Values)),
 51         volume = to_integer(lists:nth(3, Values)),

 52         avg_daily_volume = to_integer(lists:nth(4, Values)),
 53         stock_exchange = lists:nth(5, Values), % to_string
 54         market_cap = to_float(lists:nth(6, Values)), % B

 55         book_value = to_float(lists:nth(7, Values)),
 56         ebitda = to_float(lists:nth(8, Values)), % B
 57         dividend_per_share = to_float(lists:nth(9, Values)),

 58         dividend_yield = to_float(lists:nth(10, Values)),
 59         earnings_per_share = to_float(lists:nth(11, Values)),
 60         week_52_high = to_float(lists:nth(12, Values)),
 61         week_52_low = to_float(lists:nth(13, Values)),

 62         day_50_moving_avg = to_float(lists:nth(14, Values)),
 63         day_200_moving_avg = to_float(lists:nth(15, Values)),
 64         price_earnings_ratio = to_float(lists:nth(16, Values)),
 65         price_earnings_growth_ratio = to_float(lists:nth(17, Values)),

 66         price_sales_ratio = to_float(lists:nth(18, Values)),
 67         price_book_ratio = to_float(lists:nth(19, Values)),
 68         short_ratio = to_float(lists:nth(20, Values))}.
 69 

 70 url(Symbol) ->
 71     "http://finance.yahoo.com/d/quotes.csv?s=" ++ Symbol ++ "&f=l1c1va2xj1b4j4dyekjm3m4rr5p5p6s7".
 72 

 73 to_float(<<"N/A">>) ->
 74     -1;
 75 to_float(Bin) ->
 76     {Multiplier, Bin1} = case bin_ends_with(Bin, <<$B>>) of

 77         true ->
 78             {1000000000, bin_replace(Bin, <<$B>>, <<>>)};
 79         false ->
 80             case bin_ends_with(Bin, <<$M>>) of

 81                 true ->
 82                     {1000000, bin_replace(Bin, <<$M>>, <<>>)};
 83                 false ->
 84                     {1,Bin}

 85             end
 86     end,
 87     L = binary_to_list(Bin1),
 88     list_to_float(L) * Multiplier.

 89
 90
 91

Note that I am omitting some code in above listing, as I just wanted to highlight HTTP request and parsing code.

Publishing the Stock Quote

I used exmpp library to communicate with the XMPP server in Erlang. Here is the code for publishing the quotes using Bosh/XMPP protocol:

  1 %%%-------------------------------------------------------------------

  2 %%% File : quote_publisher.erl
  3 %%% Author : Shahzad Bhatti
  4 %%% Purpose : OTP server for publishing quotes

  5 %%% Created : May 8, 2010
  6 %%%-------------------------------------------------------------------
  7 -module(quote_publisher).

  8
  9 -export([
 10     start/1,
 11     start/5,
 12     stop/1]).
 13 

 14 -export([init/5]).
 15
 16 -include_lib("quote_utils.hrl").
 17
 18 -record(state, {session, jid, service=?TEST_XMPP_PUBSUB, symbol}).

 19
 20 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 21 %% APIs
 22 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

 23 start(Symbol) ->
 24     start(?TEST_XMPP_SERVER, ?TEST_XMPP_PORT, ?PRODUCER_USERNAME,
 25         ?PRODUCER_PASSWORD, Symbol).

 26
 27 start(Host, Port, User, Password, Symbol) ->
 28     spawn(?MODULE, init, [Host, Port, User, Password, Symbol]).

 29
 30 stop(Pid) ->
 31     Pid ! stop.
 32
 33 init(Host, Port, User, Password, Symbol) ->

 34     {ok, {MySession, MyJID}} = quote_utils:connect(Host, Port, User, Password),
 35     State = #state{session=MySession, jid=MyJID, symbol = Symbol},

 36     create_symbol_node(State),
 37     loop(State).
 38
 39 loop(#state{session=MySession, jid=_MyJID, service = _Service,

 40         symbol = _Symbol}=State) ->
 41     receive
 42         stop ->
 43             quote_utils:disconnect(MySession);

 44         Record = #received_packet{packet_type=message, raw_packet=_Packet} ->
 45             loop(State);
 46         Record ->

 47             loop(State)
 48     after 2000 ->
 49         publish_quote(State),
 50         loop(State)

 51     end.
 52
 53 create_symbol_node(#state{session=MySession, jid=MyJID, service = Service,

 54         symbol = Symbol}) ->
 55     IQ = exmpp_client_pubsub:create_node(Service, Symbol),
 56     PacketId = exmpp_session:send_packet(MySession, exmpp_stanza:set_sender(IQ, MyJID)),

 57     PacketId2 = erlang:binary_to_list(PacketId),
 58     receive #received_packet{id=PacketId2, raw_packet=Raw} ->

 59       case exmpp_iq:is_error(Raw) of
 60         true -> {error, Raw};
 61         _ -> ok

 62       end
 63     end.
 64
 65 publish_quote(#state{session=MySession, jid=MyJID, service = Service, symbol = Symbol}) ->

 66     Quote = yquote_client:quote(Symbol),
 67     JsonQuote = ?record_to_json(quote, Quote),
 68     M = exmpp_xml:element(?QUOTE_DATA),

 69     IQ = exmpp_client_pubsub:publish(Service, Symbol, exmpp_xml:append_cdata(M,
 70             JsonQuote)),
 71     Xml = exmpp_stanza:set_id(exmpp_stanza:set_sender(IQ, MyJID), Symbol),

 72     PacketId = exmpp_session:send_packet(MySession, exmpp_stanza:set_sender(IQ, MyJID)),
 73     PacketId2 = erlang:binary_to_list(PacketId),

 74     receive #received_packet{id=PacketId2, raw_packet=Raw} ->
 75       case exmpp_iq:is_error(Raw) of

 76         true -> error;
 77         _ -> ok
 78       end
 79     end.

 80
 81
 82

In above code, a process is created for each symbol, which periodically polls stock quote and publishes it to the XMPP node using pubsub/bosh protocol. Note that a unique node is created for each symbol and node must be created before anyone can publish or subscribe. Also, note that publish/subscribe APIs use request/ack protocol, so after sending the request, the process retrieves the acknowledgement of the request.

Here are some utility functions used by the publisher:

  1 -module(quote_utils).

  2
  3 -include_lib("quote_utils.hrl").
  4
  5 -export([
  6     init_session/2,

  7     connect/4,
  8     disconnect/1]).
  9
 10 bosh_url(Host, Port) ->

 11     "http://" ++ Host ++ ":" ++ integer_to_list(Port) ++ "/http-bind".
 12 

 13
 14 connect(Host, _Port, User, Password) ->
 15     safe_start_apps(),

 16     MySession = exmpp_session:start({1,0}),
 17     exmpp_xml:start_parser(), %% Create XMPP ID (Session Key):
 18     MyJID = exmpp_jid:make(User, Host, random),

 19     %% Create a new session with basic (digest) authentication:
 20     exmpp_session:auth_basic_digest(MySession, MyJID, Password),
 21     

 22
 23     {ok, _StreamId, _Features} = exmpp_session:connect_BOSH(MySession, bosh_url(Host, 5280), Host, []),

 24     try quote_utils:init_session(MySession, Password)
 25     catch
 26         _:Error -> io:format("got error: ~p~n", [Error]), {error, Error}

 27     end,
 28     {ok, {MySession, MyJID}}.
 29
 30 init_session(MySession, Password) ->

 31     %% Login with defined JID / Authentication:
 32     try exmpp_session:login(MySession, "PLAIN")
 33     catch

 34         throw:{auth_error, 'not-authorized'} ->
 35         %% Try creating a new user:
 36         io:format("Register~n",[]),
 37         %% In a real life client, we should trap error case here

 38         %% and print the correct message.
 39         exmpp_session:register_account(MySession, Password),
 40         %% After registration, retry to login:

 41         exmpp_session:login(MySession)
 42     end,
 43     %% We explicitely send presence:
 44     exmpp_session:send_packet(MySession, exmpp_presence:set_status(exmpp_presence:available(), "Ready to publish!!!")),

 45     ok.
 46
 47 disconnect(MySession) ->
 48     exmpp_session:stop(MySession).
 49 

 50 safe_start_apps() ->
 51     try start_apps()
 52     catch
 53         _:Error -> io:format("apps already started : ~p~n", [Error]), {error, Error}

 54     end.
 55
 56 start_apps() ->
 57     ok = application:start(exmpp),
 58     ok = application:start(crypto),
 59     ok = application:start(ssl),

 60     ok = application:start(lhttpc).
 61

Note that above code auto-registers users, which is not recommended for production use.

Javascript code using Strophe library

The web application depends on jQuery, Strophe and Strophe Pubsub. These libraries are included in docroot directory that are imported by index.html. The Strophe library and ejabbered 2.1.3 version supports cross domain scripting so that bosh service here doesn’t need to be on the same domain/port, but it must have a /crossdomain.xml policy file that allows access from wherever index.html lives. The Javascript initializes the connection parameter as follows (you would have to change Host):

   1 <script type="text/javascript">

   2     // The BOSH_SERVICE here doesn't need to be on the same domain/port, but

   3     // it must have a /crossdomain.xml policy file that allows access from

   4     // wherever crossdomain.html lives.
   5     // TODO: REPLACE <paraclete.local> with your <host-name>

   6     var HOST = 'paraclete.local';
   7     var JID = 'consumer@' + HOST;

   8     var PASSWORD = 'consumer';
   9     var BOSH_SERVICE = 'http://' + HOST + ':5280/http-bind'; //'/xmpp-httpbind'

  10     var PUBSUB = 'pubsub.' + HOST;
  11     var connection = null;

  12     var autoReconnect = true;
  13     var hasQuotes = [];
  14     var subscriptions = [];

  15
  16     function log(msg) {
  17         $('#log').append('<div></div>').append(document.createTextNode(msg));

  18     }
  19
  20     function rawInput(data) {
  21         //log('RECV: ' + data);

  22     }
  23
  24     function rawOutput(data) {
  25         //log('SENT: ' + data);

  26     }
  27     function onQuote(stanza) {
  28         //log('onQuote###### ' + stanza);

  29         try {
  30             $(stanza).find('event items item data').each(function(idx, elem) {
  31                 quote = jQuery.parseJSON($(elem).text());

  32                 //{"price":235.86,"change":-10.39,"volume":59857756,"avg_daily_volume":20775600,"stock_exchange":[78,97,115,100,97,113,78,77],"market_cap":2.146e+11,

  33                 //"book_value":43.257,"ebitda":1.5805e+10,"dividend_per_share":0.0,"dividend_yield":-1,"earnings_per_share":11.796,"week_52_high":272.46,"week_52_low":119.38,

  34                 //"day_50_moving_avg":245.206,"day_200_moving_avg":214.119,"price_earnings_ratio":20.88,"price_earnings_growth_ratio":1.05,"price_sales_ratio":4.38,

  35                 //"price_book_ratio":5.69,"short_ratio":0.7}
  36                 if (hasQuotes[quote.symbol] != undefined) {

  37                     $('price_' + quote.symbol).innerHTML = quote.price;
  38                     $('change_' + quote.symbol).innerHTML = quote.change;
  39                     $('volume_' + quote.symbol).innerHTML = quote.volume;

  40                 } else {
  41                     hasQuotes[quote.symbol] = true;
  42                     $('#quotesTable > tbody:last').append('<tr id="quote_' +

  43                         quote.symbol + '"><td>' + quote.symbol +
  44                         '</td><td id="price_' + quote.symbol + '">' + quote.price +

  45                         '</td><td id="change_' + quote.symbol + '" class="class_change_' + quote.symbol + '">' +
  46                         quote.change + '</td><td id="volume_' +

  47                         quote.symbol + '">' +
  48                         quote.volume + '</td></tr>');
  49                 }

  50
  51                 if(quote.change < 0) {
  52                     $('.class_change_' + quote.symbol).css('color', 'red');

  53                 } else {
  54                     $('.class_change_' + quote.symbol).css('color', 'green');

  55                 }
  56             });
  57         } catch (e) {
  58             log(e)

  59         }
  60         return true;
  61     }
  62 

  63     function handleSubscriptionChange (stanza) {
  64         //log("***handleSubscriptionChange Received: " + stanza);

  65     }
  66
  67     function onConnect(status) {
  68         if (status == Strophe.Status.CONNECTING) {

  69             log('Strophe is connecting.');
  70         } else if (status == Strophe.Status.CONNFAIL) {
  71             log('Strophe failed to connect.');

  72             $('#connect').get(0).value = 'connect';
  73         } else if (status == Strophe.Status.DISCONNECTING) {

  74             log('Strophe is disconnecting.');
  75         } else if (status == Strophe.Status.DISCONNECTED) {
  76             if (autoReconnect) {

  77                 log( "Streaming disconnected. Trying to reconnect...", METHODNAME );
  78                 connection.connect($('#jid').get(0).value, $('#pass').get(0).value, onConnect);
  79                 log( "Streaming reconnected.", METHODNAME );

  80             } else {
  81                 log('Strophe is disconnected.');
  82                 $('#connect').get(0).value = 'connect';

  83                 //publishEvent( "streamingDisconnected" );
  84             }
  85         } else if (status == Strophe.Status.CONNECTED) {

  86             log('Strophe is connected.');
  87             //log('QUOTE_BOT: Send a message to ' + connection.jid + ' to talk to me.');

  88             connection.addHandler(onMessage, null, 'message', null, null, null);
  89             connection.send($pres().tree());

  90             publishEvent( "streamingConnected" );
  91         }
  92     }
  93 

  94     function subscribe(symbol) {
  95         if (subscriptions[symbol]) return;
  96         try {

  97             connection.pubsub.subscribe(JID, PUBSUB, symbol, [], onQuote, handleSubscriptionChange);
  98             subscriptions[symbol] = true;
  99             log("Subscribed to " + symbol);

 100         } catch (e) {
 101             alert(e)
 102         }
 103     }
 104     function unsubscribe(symbol) {

 105         if (!subscriptions[symbol]) return;
 106         try {
 107             connection.pubsub.unsubscribe(JID, PUBSUB, symbol, handleSubscriptionChange);
 108             subscriptions[symbol] = false;

 109             log("Unsubscribed from " + symbol);
 110         } catch (e) {
 111             alert(e)
 112         }

 113     }
 114
 115     function onMessage(msg) {
 116         var to = msg.getAttribute('to');

 117         var from = msg.getAttribute('from');
 118         var type = msg.getAttribute('type');
 119         var elems = msg.getElementsByTagName('body');

 120
 121         if (type == "chat" && elems.length > 0) {
 122             var body = elems[0];

 123             log('QUOTE_BOT: I got a message from ' + from + ': ' + Strophe.getText(body));
 124             var reply = $msg({to: from, from: to, type: 'chat'}).cnode(Strophe.copyElement(body));
 125             connection.send(reply.tree());

 126             log('QUOTE_BOT: I sent ' + from + ': ' + Strophe.getText(body));
 127         }
 128         // we must return true to keep the handler alive.

 129         // returning false would remove it after it finishes.

 130         return true;
 131     }
 132
 133     $(document).ready(function () {

 134         connection = new Strophe.Connection(BOSH_SERVICE);
 135         connection.rawInput = rawInput;
 136         connection.rawOutput = rawOutput;
 137         connection.connect(JID, PASSWORD, onConnect);
 138         //connection.disconnect();

 139         $('#add_symbol').bind('click', function () {
 140             var symbol = $('#symbol').get(0).value;

 141             subscribe(symbol);
 142         });
 143     });
 144
 145 </script>
 146

When the document is loaded, the connection to the ejabberd server is established. Here is the form and table that is used to add subscription and display current quote information for the symbols:

  1 <form name='symbols'>

  2     <label for='symbol'>Symbol:</label>
  3     <input type='text' id='symbol'/>

  4     <input type='button' id='add_symbol' value='add' />

  5 </form>
  6 <hr />
  7 <div id='log'></div>

  8 <table id="quotesTable" width="600" border="2" bordercolor="#333333">

  9     <thead>
 10         <tr>
 11             <th>Symbol</th>

 12             <th>Price</th>
 13             <th>Change</th>
 14             <th>Volume</th>

 15         </tr>
 16     </thead>
 17     <tbody>
 18     </tbody>

 19 </table>
 20

When the form is submitted, it calls subscribe method, which in turn sends request to the ejabbered server for subscription. When a new quote is received, it calls onQuote function, which inserts a row in the table when a new symbol is added or updates the quote information if it already exists.

Conclusion

The ejabberd, XMPP, exmpp, Bosh and Strophe provides a robust and mature solution for messaging and are especially suitable for web applications that want to build highly scalable and interactive applications. Though, above code is fairly simple, but same design principles can be used to support large number of stock quotes updates. As, we need to send stock quotes from tens of thousands symbols for every tick within a fraction of a second, the Erlang provides very scalable solution, where each symbol is simply served by an Erlang process. Finally, I am still learning more about Ejabberd’s clustering, security, and other features so that it can truly survive the production load, so I would love to hear any feedback you might have with similar systems.

References

Posted by admin
Filed in Erlang
No Comments »

First Arduino project

April 29th, 2010

I recently bout Arduino Starter Kit that comes with Arduino Duemilanove ATmega328 and a bunch of LEDs, buttons, resisters and wires. I then downloaded the Arduino software from http://www.arduino.cc/en/Main/Software. I downloaded the Mac OS X version and installed it on my Mac Pro laptop. Once the software was installed, I launched the software and connected USB cable to the Arduino board as below:

Arudino software comes with a number of small applications and I tried their blink application by selecting File->Examples->Digital->Blink from the top menu. I then selected my board by selecting Tools->Board->ATmega328. Then I tried to look for USB connection under Tools->Serial Board, but didn’t see any. I googled for Arduino with Mac OS, and found USB driver, it required reboot so after reboot I was able to see it, e.g.

I inserted the LED to pin 13 that has builtin resistor, where the longer leg with positive charge went into the hole 13 and short leg went into the ground hole next to it. I then used the software to upload the application to the Arduino, e.g.

Voila, I was able to see the LED blinking, success.

Posted by admin
Filed in Arduino
No Comments »

Favorite fifteen tips from “Rework” book by Jason Fried and DHH

April 24th, 2010

I have been a long admirer of Jason Fried of 37Signals and read his first book Getting Real. Jason along with DHH have put forth many of the ideas from that book along with other ideas from their blog Signal vs. Noise into a new book Rework. I just finished reading it and though it reiterates many ideas from the earlier book “Getting Real” and their blogs, it’s worth re-reading those ideas as many of business companies today still runs on old fallacies. The book consists of thirteen sections and over eighty ideas, here are my favorite ideas from the book:

Failure is not a rite of passage


I have heared the advice from startup folks about “Fail early and fail often.” On the contrary, this book shows people who learn from mistakes will make new mistakes, instead success shows what actually works. Another related avice in the book is “Reason to quit”, which shows when you can quit and choose something else. When I read Founders at Work: Stories of Startups’ Early Days, it also showed that most startups don’t stick to their original ideas and move to other ideas based on early feedback.

Planning is Guessing


This is related to another advice from the book “Your estimates suck” as Planning and Estimation is hard especially in software business. I have written about Software Estimation in my earlier blogs, however most places still equate estimates with commitments. Jason and DHH reminds us again that estimates are just guesses that were made based on the best information available at the time.

Workaholism


This is another unorthodox advice that is contradictory to how most software projects are run. Most companies measure workers’ dedication on how many hours he/she put even when they are not actually producing. This is also common when managers treat estimates as commitments and refuse to admit reality when things change. We are all familiar with iron triangle of schedule/cost/functionality or sometime referred to as cost/quality/schedule or cost/resourcs/schedule. Often business folks are unwilling to change schedule and functionality, which often requires working late hours. This is also related to Heroism, which I have blogged before and go to sleep, as workholism can result in sleep deprivation, which reduces creativity and productivity.

Scratch your own itch

Most successful businesses started with hobbies or personal interests or problems and there are tons of examples of this. This advice is also related to eat your own dog food, though not mentioned in this book.

Start making something


Jason and DHH reminds us another great point that ideas are cheap and the real question is how well you execute them.

Draw a line in the sand


One of the key characteristics of Ruby on Rails software that DHH produced is having strong opinions that limits variations. Similarly, 37Signals is known for their simple design and limited features. You can differentiate yourself from others by standing for something.

Outside money is Plan Z


Both DHH and Jason often talked about downside of getting money from venture capitalists and I agree that these days you can start most software startups with minimal money and raising money can be very distracting. Another related tip that “building a flip is building to flop”, which is often what startup founders hope to get out.

Start at the epicenter


This book advices you to focus on your core product. Though, this book briefly mentiosn this topic but there is a great presentation of Video of Geoffrey Moore at Business of Software 2009 that talks about similar topic. This advice is also reated to other tips from the book such as “don’t copy”, “decommoditize your product”, “focus on you instead of they”, i.e., focus on your core strengths and not your competitors.

Focus on what won’t change


This is great advice for building business that will last. I remember when I started working at Amazon, we were told the core values of Amazon that included having a large selection, cheap prices, customer service and everything we built started from outside-in focus, i.e., it started with customers.

Get it out here


This is similar to common advice from the startup and agile community, i.e. release early and release often.

Interruption is the enemy of productivity


More and more research is showing that our brain can’t focus on onething at a time, and constant interruption and multi-tasking hampers your productivity. This is also somewhat related to office space is setup as many agile practices encourage more open space with pair programming and I have found that it prevents concentration. I found that private office pattern offered from Organizational Patterns of Agile Software Development provides less interruption.

Meetings are toxic


This is another hallmark idea of 37Signals and the book contains a number of tips on making your productive such as fixed time, fewer people, clear agenda, beginning with a specific problem and ending with action items and making someone responsible for them.

Good enough is fine


37Signals is known for their simple design and fewer features. This is related other advice in the book such as “embrace the constraints”, “throw less at the problem”, “underdo your competitor”, “say no” and “be a curator”. When you have limited resources, you can become more creative. Also, you are better off building half a product, not a half assed product.

Make tiny decisions


The authors encourage to make tiny decisions as big decisions are hard to make and hard to change. This advice is related to other tips such as “decisions are progress”, which encourages you to always make progress and “quick wins”, which encourages you to build momentum by accomplishing small tasks.

Build an audience


The authors encourage to build audience that come back to you by writing blogs, tweets and speaking. This is also reated to “sell your by-products”, “emulate chefs”, “emulate drug dealers” and “out-teach your competitors”.

Conclusion

Though, I skipped many gems of advice on hiring, culture and marketing but I suggest you read the book to build long lasting and successful business.

Posted by admin
Filed in Business
No Comments »

Smarter Email appender for Log4j with support of duplicate-removal, summary-report and JMX

March 17th, 2010

I have been using SMTPAppender for a while to notify developers when something breaks on the production site and for most part it works well. However, due to some misconfiguration or service crash it can result in large number of emails. I was struck by similar problem at work when my email box suddently got tons of emails from the production site. So I decided to write a bit intelligent email appender. My goals for the appender were:

  • Throttle emails based on some configured time
  • Remove duplicate emails
  • Support JMX for dynamic configuration
  • Provide summary report with count of errors and their timings

I created FilteredSMTPAppender class that extends SMTPAppender. The FilteredSMTPAppender defines a nested class Stats for keeping track of errors. For each unique exception, it creates an instance of Stats, that stores the first and last occurrence of this exception as well as count. The Stats class uses hash of stack trace to identify unique exceptions, however it ignores first line, which often stores some dynamic information. FilteredSMTPAppender registers iteslf as MBean so that it can be configured at runtime. It overrides append method to capture the event and overrides checkEntryConditions to add filtering. It also changes the layout so that the summary count of error messages are added to the footer of email message.

The FilteredSMTPAppender uses a number of helper classes such as ServiceJMXBeanImpl for MBean definition, LRUSortedList to keep fixed cache of exceptions. Here is listing of LRUSortedList and ServiceJMXBeanImpl.

Listing of FilteredSMTPAppender.java

   1 package com.plexobject.log;

   2
   3 import java.beans.PropertyChangeEvent;
   4 import java.beans.PropertyChangeListener;
   5 import java.util.Comparator;

   6 import java.util.Date;
   7
   8 import javax.mail.MessagingException;
   9 

  10 import org.apache.commons.lang.builder.EqualsBuilder;
  11 import org.apache.commons.lang.time.FastDateFormat;
  12
  13 import org.apache.log4j.Layout;

  14 import org.apache.log4j.net.SMTPAppender;
  15 import org.apache.log4j.spi.LoggingEvent;
  16
  17 import com.plexobject.jmx.JMXRegistrar;

  18 import com.plexobject.jmx.impl.ServiceJMXBeanImpl;
  19 import com.plexobject.metrics.Metric;
  20 import com.plexobject.metrics.Timer;

  21 import com.plexobject.util.Configuration;
  22 import com.plexobject.util.LRUSortedList;
  23
  24 public class FilteredSMTPAppender extends SMTPAppender {

  25
  26     private static final String SMTP_FILTER_MIN_DUPLICATE_INTERVAL_SECS = "smtp.filter.min.duplicate.interval.secs";
  27     private static final int MAX_STATS = Configuration.getInstance().getInteger("smtp.filter.max", 100);

  28     private static int MIN_DUPLICATE_EMAILS_INTERVAL = Configuration.getInstance().getInteger(SMTP_FILTER_MIN_DUPLICATE_INTERVAL_SECS,
  29             60); // 1 minute
  30     private static final Date STARTED = new Date();

  31     private static final FastDateFormat DATE_FMT = FastDateFormat.getInstance("MM/dd/yy HH:mm");
  32
  33     final static class Stats implements Comparable<Stats> {

  34
  35         final int checksum;
  36         final long firstSeen;

  37         long lastSeen;
  38         long lastSent;
  39         int numSeen;

  40         int numEmails;
  41
  42         Stats(LoggingEvent event) {
  43             StringBuilder sb = new StringBuilder();

  44             String[] trace = event.getThrowableStrRep();
  45             for (int i = 1; i < trace.length && i < 20; i++) { // top 20 lines

  46                 // of trace
  47                 sb.append(trace[i].trim());
  48             }
  49             this.checksum = sb.toString().hashCode();

  50             firstSeen = lastSeen = System.currentTimeMillis();
  51             numSeen = 1;
  52         }
  53
  54         boolean check() {

  55             long current = System.currentTimeMillis();
  56             long elapsed = current - lastSent;
  57
  58             numSeen++;

  59             lastSeen = current;
  60
  61             if (elapsed > MIN_DUPLICATE_EMAILS_INTERVAL * 1000) {
  62                 lastSent = current;

  63                 numEmails++;
  64                 return true;
  65             } else {

  66                 return false;
  67             }
  68         }
  69 

  70         @Override
  71         public boolean equals(Object object) {
  72             if (!(object instanceof Stats)) {

  73                 return false;
  74             }
  75             Stats rhs = (Stats) object;
  76             return new EqualsBuilder().append(this.checksum, rhs.checksum).isEquals();

  77
  78         }
  79
  80         @Override
  81         public int hashCode() {

  82             return checksum;
  83         }
  84
  85         @Override

  86         public String toString() {
  87             return " (" + checksum + ") occurred " + numSeen + " times, " + numEmails + " # of emails, first @" + DATE_FMT.format(new Date(firstSeen)) + ", last @" + DATE_FMT.format(new Date(lastSeen)) + " since server started @" + DATE_FMT.format(STARTED);

  88         }
  89
  90         @Override
  91         public int compareTo(Stats other) {

  92             return checksum - other.checksum;
  93         }
  94     }
  95 

  96     final static class StatsCmp implements Comparator<Stats> {
  97 

  98         @Override
  99         public int compare(Stats first, Stats second) {
 100             return first.checksum - second.checksum;

 101         }
 102     }
 103     private static final LRUSortedList<Stats> STATS_LIST = new LRUSortedList<Stats>(

 104             MAX_STATS, new StatsCmp());
 105     private LoggingEvent event;
 106     private ServiceJMXBeanImpl mbean;
 107     private Layout layout;

 108
 109     public FilteredSMTPAppender() {
 110         mbean = JMXRegistrar.getInstance().register(getClass());
 111         mbean.addPropertyChangeListener(new PropertyChangeListener() {
 112 

 113             @Override
 114             public void propertyChange(PropertyChangeEvent event) {
 115                 try {
 116                     if (event != null && SMTP_FILTER_MIN_DUPLICATE_INTERVAL_SECS.equalsIgnoreCase(event.getPropertyName())) {

 117                         MIN_DUPLICATE_EMAILS_INTERVAL = Integer.parseInt((String) event.getNewValue());
 118                     }
 119                 } catch (Exception e) {
 120                     e.printStackTrace();
 121                 }

 122             }
 123         });
 124
 125     }
 126
 127     public void append(LoggingEvent event) {

 128         this.event = event;
 129         if (layout == null) {
 130             layout = getLayout();
 131         }

 132         super.append(event);
 133     }
 134
 135     protected boolean checkEntryConditions() {
 136         final Timer timer = Metric.newTimer(getClass().getSimpleName() + ".checkEntryConditions");

 137         try {
 138             boolean check = true;
 139             if (event != null) {

 140                 Stats newStats = new Stats(event);
 141                 Stats stats = STATS_LIST.get(newStats);
 142                 if (stats == null) {
 143                     stats = newStats;

 144                     STATS_LIST.add(stats);
 145                 } else {
 146                     check = stats.check();
 147                 }
 148                 if (check) {

 149                     setMessageFooter(stats);
 150                 }
 151             }
 152             return check && super.checkEntryConditions();

 153         } finally {
 154             timer.stop();
 155         }
 156     }
 157 

 158     private void setMessageFooter(Stats stats) {
 159         String message = event.getMessage().toString();
 160
 161         final String footer = "\n\n-------------------------\n" + message + " - " + stats;

 162
 163         if (layout != null) {
 164             setLayout(new Layout() {
 165 

 166                 @Override
 167                 public void activateOptions() {
 168                     layout.activateOptions();
 169
 170                 }

 171
 172                 @Override
 173                 public String format(LoggingEvent evt) {
 174                     return layout.format(evt);
 175                 }

 176
 177                 @Override
 178                 public String getFooter() {
 179                     return footer;
 180                 }

 181
 182                 @Override
 183                 public boolean ignoresThrowable() {
 184                     return layout.ignoresThrowable();

 185                 }
 186             });
 187         }
 188     }
 189 }
 190 

 191

Listing of ServiceJMXBeanImpl.java

   1 package com.plexobject.util;

   2
   3 import java.util.ArrayList;
   4 import java.util.Collection;
   5 import java.util.Collections;

   6 import java.util.Comparator;
   7 import java.util.Iterator;
   8 import java.util.List;

   9 import java.util.ListIterator;
  10
  11 import org.apache.log4j.Logger;
  12 

  13
  14 public class LRUSortedList<T> implements List<T> {

  15     private static final Logger LOGGER = Logger.getLogger(LRUSortedList.class);
  16     private final int max;

  17     private final Comparator<T> comparator;
  18
  19     private final List<Pair<Long, T>> list = new ArrayList<Pair<Long, T>>();

  20     private final List<Pair<Long, Integer>> timestamps = new ArrayList<Pair<Long, Integer>>();

  21
  22     // comparator to sort by timestamp
  23     private static final Comparator<Pair<Long, Integer>> CMP = new Comparator<Pair<Long, Integer>>() {

  24         @Override
  25         public int compare(Pair<Long, Integer> first, Pair<Long, Integer> second) {

  26             if (first.getFirst() < second.getFirst()) {
  27                 return -1;
  28             } else if (first.getFirst() > second.getFirst()) {

  29                 return 1;
  30             } else {
  31                 return 0;

  32             }
  33         }
  34     };
  35
  36     public LRUSortedList(int max, Comparator<T> comparator) {

  37         this.max = max;
  38         this.comparator = comparator;
  39     }
  40 

  41     @Override
  42     public boolean add(T e) {
  43         if (list.size() > max) {

  44             removeOldest();
  45         }
  46         // add object
  47         long timestamp = System.nanoTime();

  48         int insertionIdx = Collections.binarySearch(this, e, comparator);
  49         if (insertionIdx < 0) {// not found

  50             insertionIdx = (-insertionIdx) - 1;
  51             list.add(insertionIdx, new Pair<Long, T>(timestamp, e));
  52         } else {

  53             // found
  54             list.set(insertionIdx, new Pair<Long, T>(timestamp, e));
  55         }

  56
  57         // as timestamps are sorted, we just remove the oldest (first)
  58         if (timestamps.size() > max) {

  59             timestamps.remove(0);
  60         }
  61         // update timestamp
  62         Pair<Long, Integer> t = new Pair<Long, Integer>(timestamp, insertionIdx);

  63         timestamps.add(t);
  64         return true;
  65     }
  66 

  67     @Override
  68     public void add(int index, T element) {
  69         throw new UnsupportedOperationException(

  70                 "can't add element at arbitrary index, must use add to keep sorted order");
  71     }
  72
  73     @Override

  74     public boolean addAll(Collection<? extends T> c) {
  75         for (T e : c) {

  76             add(e);
  77         }
  78         return c.size() > 0;
  79     }

  80
  81     @Override
  82     public boolean addAll(int index, Collection<? extends T> c) {

  83         throw new UnsupportedOperationException(
  84                 "can't add element at arbitrary index, must use addAll to keep sorted order");
  85     }

  86
  87     @Override
  88     public void clear() {
  89         list.clear();

  90     }
  91
  92     @SuppressWarnings("unchecked")
  93     @Override

  94     public boolean contains(Object e) {
  95         if (e == null) {
  96             return false;

  97         }
  98         try {
  99             return Collections.binarySearch(this, (T) e, comparator) >= 0;

 100         } catch (ClassCastException ex) {
 101             LOGGER.error("Unexpected type for contains "
 102                     + e.getClass().getName() + ": " + e);

 103             return false;
 104         }
 105     }
 106
 107     @Override

 108     public boolean containsAll(Collection<?> c) {
 109         for (Object e : c) {
 110             if (!contains(e)) {

 111                 return false;
 112             }
 113         }
 114         return true;

 115     }
 116
 117     @Override
 118     public T get(int index) {
 119         Pair<Long, T> e = list.get(index);

 120         return e != null ? e.getSecond() : null;
 121     }
 122
 123     public T get(Object e) {

 124         int ndx = indexOf(e);
 125         if (ndx >= 0) {
 126             return get(ndx);
 127         }

 128         return null;
 129     }
 130
 131     @SuppressWarnings("unchecked")
 132     @Override

 133     public int indexOf(Object e) {
 134         try {
 135             return Collections.binarySearch(this, (T) e, comparator);

 136         } catch (ClassCastException ex) {
 137             LOGGER.error("Unexpected type for get " + e.getClass().getName()
 138                     + ": " + e);

 139             return -1;
 140         }
 141     }
 142
 143     @Override
 144     public boolean isEmpty() {

 145         return list.isEmpty();
 146     }
 147
 148     @Override
 149     public Iterator<T> iterator() {

 150         final Iterator<Pair<Long, T>> it = list.iterator();
 151         return new Iterator<T>() {

 152
 153             @Override
 154             public boolean hasNext() {
 155                 return it.hasNext();

 156             }
 157
 158             @Override
 159             public T next() {
 160                 Pair<Long, T> e = it.next();

 161                 return e.getSecond();
 162             }
 163
 164             @Override
 165             public void remove() {

 166                 it.remove();
 167             }
 168         };
 169     }
 170
 171     @Override

 172     public int lastIndexOf(Object o) {
 173         for (int i = list.size() - 1; i >= 0; i--) {
 174             T e = get(i);

 175             if (e.equals(o)) {
 176                 return i;
 177             }
 178         }
 179         return -1;

 180     }
 181
 182     @Override
 183     public ListIterator<T> listIterator() {
 184         final ListIterator<Pair<Long, T>> it = list.listIterator();

 185         return buildListIterator(it);
 186     }
 187
 188     @Override
 189     public ListIterator<T> listIterator(int index) {

 190         final ListIterator<Pair<Long, T>> it = list.listIterator(index);
 191         return buildListIterator(it);
 192     }

 193
 194     @SuppressWarnings("unchecked")
 195     @Override
 196     public boolean remove(Object e) {

 197         try {
 198             int ndx = Collections.binarySearch(this, (T) e, comparator);
 199             if (ndx >= 0) {

 200                 remove(ndx);
 201                 return true;
 202             } else {
 203                 return false;

 204             }
 205
 206         } catch (ClassCastException ex) {
 207             LOGGER.error("Unexpected type for remove " + e.getClass().getName()

 208                     + ": " + e);
 209             return false;
 210         }
 211     }

 212
 213     @Override
 214     public T remove(int index) {
 215         Pair<Long, T> e = list.remove(index);

 216         Pair<Long, Integer> t = new Pair<Long, Integer>(e.getFirst(), 0);
 217
 218         int insertionIdx = Collections.binarySearch(timestamps, t, CMP);

 219         if (insertionIdx >= 0) {
 220             timestamps.remove(insertionIdx);
 221         }
 222         return e != null ? e.getSecond() : null;

 223     }
 224
 225     @Override
 226     public boolean removeAll(Collection<?> c) {

 227         boolean all = true;
 228         for (Object e : c) {
 229             all = all && remove(e);

 230         }
 231         return all;
 232     }
 233
 234     @Override
 235     public boolean retainAll(Collection<?> c) {

 236         boolean changed = false;
 237         Iterator<?> it = c.iterator();
 238         while (it.hasNext()) {

 239             Object e = it.next();
 240             if (!contains(e)) {
 241                 it.remove();
 242                 changed = true;
 243             }

 244         }
 245         return changed;
 246     }
 247
 248     @Override

 249     public T set(int index, T element) {
 250         throw new UnsupportedOperationException();
 251     }

 252
 253     @Override
 254     public int size() {
 255         return list.size();

 256     }
 257
 258     @Override
 259     public List<T> subList(int fromIndex, int toIndex) {

 260         List<T> tlist = new ArrayList<T>();
 261         List<Pair<Long, T>> plist = list.subList(fromIndex, toIndex);

 262         for (Pair<Long, T> e : plist) {
 263             tlist.add(e.getSecond());
 264         }
 265         return tlist;

 266     }
 267
 268     @Override
 269     public Object[] toArray() {
 270         return subList(0, list.size()).toArray();

 271     }
 272
 273     @SuppressWarnings("hiding")
 274     @Override
 275     public <T> T[] toArray(T[] a) {

 276         return subList(0, list.size()).toArray(a);
 277     }
 278
 279     @Override
 280     public String toString() {

 281         StringBuilder sb = new StringBuilder();
 282         Iterator<T> it = iterator();
 283         while (it.hasNext()) {

 284             sb.append(it.next() + ", ");
 285         }
 286         return sb.toString();
 287     }
 288 

 289     private void removeOldest() {
 290         timestamps.remove(timestamps.size() - 1);
 291     }
 292
 293     private ListIterator<T> buildListIterator(

 294             final ListIterator<Pair<Long, T>> it) {
 295         return new ListIterator<T>() {

 296
 297             @Override
 298             public void add(T e) {
 299                 it.add(new Pair<Long, T>(System.nanoTime(), e));

 300             }
 301
 302             @Override
 303             public boolean hasNext() {
 304                 return it.hasNext();

 305
 306             }
 307
 308             @Override
 309             public boolean hasPrevious() {

 310                 return it.hasPrevious();
 311
 312             }
 313
 314             @Override
 315             public T next() {

 316                 Pair<Long, T> e = it.next();
 317                 return e.getSecond();
 318             }
 319
 320             @Override

 321             public int nextIndex() {
 322                 return it.nextIndex();
 323
 324             }

 325
 326             @Override
 327             public T previous() {
 328                 Pair<Long, T> e = it.previous();

 329                 return e.getSecond();
 330             }
 331
 332             @Override
 333             public int previousIndex() {

 334                 return it.previousIndex();
 335
 336             }
 337
 338             @Override
 339             public void remove() {

 340                 it.remove();
 341
 342             }
 343
 344             @Override
 345             public void set(T e) {

 346                 it.set(new Pair<Long, T>(System.nanoTime(), e));
 347
 348             }
 349         };
 350     }

 351
 352 }
 353
 354

Listing of LRUSortedList.java

   1 package com.plexobject.jmx.impl;

   2
   3 import java.beans.PropertyChangeListener;
   4 import java.beans.PropertyChangeSupport;
   5 import java.util.Map;

   6 import java.util.concurrent.ConcurrentHashMap;
   7 import java.util.concurrent.atomic.AtomicLong;
   8
   9 import javax.management.AttributeChangeNotification;

  10 import javax.management.MBeanNotificationInfo;
  11 import javax.management.Notification;
  12 import javax.management.NotificationBroadcasterSupport;

  13 import javax.management.NotificationListener;
  14
  15 import org.apache.commons.lang.builder.EqualsBuilder;
  16 import org.apache.commons.lang.builder.HashCodeBuilder;

  17 import org.apache.commons.lang.builder.ToStringBuilder;
  18 import org.apache.log4j.Logger;
  19
  20 import com.plexobject.jmx.ServiceJMXBean;

  21 import com.plexobject.metrics.Metric;
  22 import com.plexobject.util.TimeUtils;
  23
  24 public class ServiceJMXBeanImpl extends NotificationBroadcasterSupport

  25         implements ServiceJMXBean, NotificationListener {
  26     private static final Logger LOGGER = Logger
  27             .getLogger(ServiceJMXBeanImpl.class);

  28     private Map<String, String> properties = new ConcurrentHashMap<String, String>();
  29     private final PropertyChangeSupport pcs = new PropertyChangeSupport(this);

  30
  31     private final String serviceName;
  32     private AtomicLong totalErrors;

  33     private AtomicLong totalRequests;
  34
  35     private AtomicLong sequenceNumber;
  36     private String state;

  37
  38     public ServiceJMXBeanImpl(final String serviceName) {
  39         this.serviceName = serviceName;

  40         this.totalErrors = new AtomicLong();
  41         this.totalRequests = new AtomicLong();
  42         this.sequenceNumber = new AtomicLong();

  43     }
  44
  45     @Override
  46     public double getAverageElapsedTimeInNanoSecs() {

  47         return Metric.getMetric(getServiceName())
  48                 .getAverageDurationInNanoSecs();
  49     }
  50 

  51     public String getProperty(final String name) {
  52         return properties.get(name);
  53     }

  54
  55     public void setProperty(final String name, final String value) {

  56         final String oldValue = properties.put(name, value);
  57         final Notification notification = new AttributeChangeNotification(this,

  58                 sequenceNumber.incrementAndGet(), TimeUtils
  59                         .getCurrentTimeMillis(), name + " changed", name,
  60                 "String", oldValue, value);
  61         sendNotification(notification);

  62         handleNotification(notification, null);
  63     }
  64
  65     @Override

  66     public String getServiceName() {
  67         return serviceName;
  68     }
  69 

  70     @Override
  71     public long getTotalDurationInNanoSecs() {
  72         return Metric.getMetric(getServiceName()).getTotalDurationInNanoSecs();

  73     }
  74
  75     @Override
  76     public long getTotalErrors() {

  77         return totalErrors.get();
  78     }
  79
  80     public void incrementError() {

  81         final long oldErrors = totalErrors.getAndIncrement();
  82         final Notification notification = new AttributeChangeNotification(this,

  83                 sequenceNumber.incrementAndGet(), TimeUtils
  84                         .getCurrentTimeMillis(), "Errors changed", "Errors",
  85                 "long", oldErrors, oldErrors + 1);

  86         sendNotification(notification);
  87     }
  88
  89     @Override
  90     public long getTotalRequests() {

  91         return totalRequests.get();
  92     }
  93
  94     public void incrementRequests() {

  95         final long oldRequests = totalRequests.getAndIncrement();
  96         final Notification notification = new AttributeChangeNotification(this,

  97                 sequenceNumber.incrementAndGet(), TimeUtils
  98                         .getCurrentTimeMillis(), "Requests changed",
  99                 "Requests", "long", oldRequests, oldRequests + 1);

 100         sendNotification(notification);
 101     }
 102
 103     @Override
 104     public MBeanNotificationInfo[] getNotificationInfo() {
 105         String[] types = new String[] { AttributeChangeNotification.ATTRIBUTE_CHANGE };

 106         String name = AttributeChangeNotification.class.getName();
 107         String description = "An attribute of this MBean has changed";
 108         MBeanNotificationInfo info = new MBeanNotificationInfo(types, name,

 109                 description);
 110
 111         return new MBeanNotificationInfo[] { info };
 112     }
 113 

 114     @Override
 115     public String getState() {
 116         return state;
 117     }
 118 

 119     /**
 120      * @param state
 121      *            the state to set

 122      */
 123     public void setState(String state) {
 124         this.state = state;
 125     }

 126
 127     /**
 128      * @see java.lang.Object#equals(Object)

 129      */
 130     @Override
 131     public boolean equals(Object object) {
 132         if (!(object instanceof ServiceJMXBeanImpl)) {

 133             return false;
 134         }
 135         ServiceJMXBeanImpl rhs = (ServiceJMXBeanImpl) object;
 136         return new EqualsBuilder().append(this.serviceName, rhs.serviceName)

 137                 .isEquals();
 138     }
 139
 140     /**
 141      * @see java.lang.Object#hashCode()

 142      */
 143     @Override
 144     public int hashCode() {
 145         return new HashCodeBuilder(786529047, 1924536713).append(

 146                 this.serviceName).toHashCode();
 147     }
 148
 149     /**
 150      * @see java.lang.Object#toString()

 151      */
 152     @Override
 153     public String toString() {
 154         return new ToStringBuilder(this)

 155                 .append("serviceName", this.serviceName).append("totalErrors",
 156                         this.totalErrors).append("totalRequests",
 157                         this.totalRequests).append("totalRequests",

 158                         this.totalRequests).append("state", this.state).append(
 159                         "properties", this.properties).toString();
 160     }

 161
 162     public void addPropertyChangeListener(PropertyChangeListener pcl) {
 163         pcs.addPropertyChangeListener(pcl);
 164     }
 165 

 166     public void removePropertyChangeListener(PropertyChangeListener pcl) {
 167         pcs.removePropertyChangeListener(pcl);
 168
 169     }
 170 

 171     @Override
 172     public void handleNotification(Notification notification, Object handback) {
 173         LOGGER.info("Received notification: ClassName: "
 174                 + notification.getClass().getName() + ", Source: "

 175                 + notification.getSource() + ", Type: "
 176                 + notification.getType() + ", tMessage: "
 177                 + notification.getMessage());
 178         if (notification instanceof AttributeChangeNotification) {

 179             AttributeChangeNotification acn = (AttributeChangeNotification) notification;
 180             pcs.firePropertyChange(acn.getAttributeName(), acn.getOldValue(),
 181                     acn.getNewValue());
 182
 183         }
 184     }

 185 }
 186
 187

Testing

Finally, here is how you can test this filter:

  1 package com.plexobject;

  2
  3 import java.net.InetAddress;
  4 import java.util.Date;
  5 

  6 import org.apache.log4j.Logger;
  7 import org.apache.log4j.PatternLayout;
  8 import org.apache.log4j.net.SMTPAppender;

  9
 10 import com.plexobject.log.FilteredSMTPAppender;
 11
 12 public class Main {

 13     private static final Logger LOGGER = Logger.getLogger(Main.class);
 14     public static void main(String[] args) {

 15         SMTPAppender appender = new FilteredSMTPAppender();
 16         try {
 17             appender.setTo("bhatti@xxx.com");
 18             appender.setFrom("bhatti@xxx.com");

 19             appender.setSMTPHost("smtp.xxx.net");
 20             appender.setLocationInfo(true);
 21             appender.setSubject("Error from " + InetAddress.getLocalHost());
 22 

 23             appender.setLayout(new PatternLayout());
 24             appender.activateOptions();
 25             LOGGER.addAppender(appender);
 26         } catch (Exception e) {

 27             LOGGER.error("Failed to register smtp appender", e);
 28         }
 29         while (true) {
 30             try {

 31                 throw new Exception("throwing exception at " + new Date());
 32             } catch (Exception e) {

 33                 LOGGER.error("Logging error at " + new Date(), e);
 34             }
 35             try {

 36                 Thread.sleep(1000);
 37             } catch (InterruptedException e) {
 38                 Thread.interrupted();
 39             }
 40         }

 41     }
 42 }
 43
 44

Above code simulates error generation every second, but it sends email based on the throttling level defined in the configuration. Obviously you can use log4j properties file to define all this configuration, e.g.

<!– Send email when error happens –>
<appender name=”APP-EMAIL” class=”com.plexobject.log.FilteredSMTPAppender”>
<param name=”BufferSize” value=”256″ />
<param name=”SMTPHost” value=”smtp.xxx.net” />
<param name=”From” value=”bhatti@xxx.com” />
<param name=”To” value=”bhatti@xxx.com” />
<param name=”Subject” value=”Production Error” />
<layout class=”org.apache.log4j.PatternLayout”>
<param name=”ConversionPattern”
value=”[%d{ISO8601}]%n%n%-5p%n%n%c%n%n%m%n%n” />
</layout>

<filter class=”org.apache.log4j.varia.StringMatchFilter”>
<param name=”StringToMatch” value=”My Error”/>
<param name=”AcceptOnMatch” value=”false” />
</filter>
</appender>

Summary

I am skipping other classes, but you can download entire code from FilteredSMTPAppender.zip. This solution seems to be working from me but feel free to share your experience with similar problems.

Posted by admin
Filed in Computing
No Comments »

A few recipes for reprocessing messages in Dead-Letter-Queue using ActiveMQ

February 3rd, 2010

Messaging based asynchronous processing is a key component of any complexed software especially in transactional environment. There are a number of solutions that provide high performance and reliable messaging in Java space such as ActiveMQ, FUSE broker, JBossMQ, SonicMQ, Weblogic, Websphere, Fiorano, etc. These providers support JMS specification, which provides abstraction for queues, message providers and message consumers. In this blog, I will go over some recipes for recovering messages from dead letter queue when using ActiveMQ.

What is Dead Letter Queue

Generally, when a consumer fails to process a message within a transaction or does not send acknowledgement back to the broker, the message is put back to the queue. The message is then delivered upto certain number of times based on configuration and finally the message is put to dead letter queue when that limit is exceeded. The ActiveMQ documentation recommends following settings for defining dead letter queues:

 <broker...>
   <destinationPolicy>
     <policyMap>
       <policyEntries>
         <!-- Set the following policy on all queues using the '>' wildcard -->
         <policyEntry queue=">">
           <deadLetterStrategy>
             <individualDeadLetterStrategy
               queuePrefix="DLQ." useQueueForQueueMessages="true" />
           </deadLetterStrategy>
         </policyEntry>
       </policyEntries>
     </policyMap>
   </destinationPolicy>
   ...
 </broker>

and you can control redlivery policy as follows:

 RedeliveryPolicy policy = connection.getRedeliveryPolicy();
 policy.setInitialRedeliveryDelay(500);
 policy.setBackOffMultiplier(2);
 policy.setUseExponentialBackOff(true);
 policy.setMaximumRedeliveries(2);

It is important that you create dlq per queue, otherwise ActiveMQ puts them into a single dead letter queue.

Getting the QueueViewMBean Handle

ActiveMQ provides QueueViewMBean to invoke administration APIs on the queues. The easiest way to get this handle is to use BrokerFacadeSupport class, which is extended by RemoteJMXBrokerFacade and LocalBrokerFacade. You can use RemoteJMXBrokerFacade if you are connecting to remote ActiveMQ server, e.g. here is Spring configuration for setting it up:

     <bean id="brokerQuery" class="org.apache.activemq.web.RemoteJMXBrokerFacade" autowire="constructor" destroy-method="shutdown">
             <property name="configuration">
             <bean class="org.apache.activemq.web.config.SystemPropertiesConfiguration"/>
         </property>
             <property name="brokerName"><null/></property>
     </bean>

Alternatively, you can use LocalBrokerFacade if you are running embedded ActiveMQ server, e.g. below is Spring configuration for it:

     <bean id="brokerQuery" class="org.apache.activemq.web.LocalBrokerFacade" autowire="constructor" scope="prototype"/>

Getting number of messages from the queue

Once you got handle to QueueViewMBean, you can use following API to find the number of messages in the queue:

 1     public long getQueueSize(final String dest) {

 2         try {
 3             return brokerQuery.getQueue(dest).getQueueSize();
 4         } catch (Exception e) {
 5             throw new RuntimeException(e);

 6         }
 7     }
 8

Copying Messages using JMS APIs

The JMS specification provides APIs to browse queue in read mode and then you can send the messages to another queue, e.g.

  1 import java.util.Enumeration;

  2
  3 import javax.jms.Connection;
  4 import javax.jms.ConnectionFactory;
  5 import javax.jms.JMSException;

  6 import javax.jms.Message;
  7 import javax.jms.Queue;
  8 import javax.jms.QueueBrowser;

  9 import javax.jms.Session;
 10 import javax.jms.TextMessage;
 11 import javax.management.openmbean.CompositeData;
 12 

 13 import org.apache.activemq.broker.jmx.QueueViewMBean;
 14 import org.apache.activemq.web.BrokerFacadeSupport;
 15 import org.springframework.beans.factory.annotation.Autowired;
 16 import org.springframework.jms.core.BrowserCallback;

 17 import org.springframework.jms.core.JmsTemplate;
 18 import org.springframework.jms.core.MessageCreator;
 19
 20 public class DlqReprocessor {

 21     @Autowired
 22     private JmsTemplate jmsTemplate;
 23
 24     @Autowired
 25     BrokerFacadeSupport brokerQuery;
 26 

 27     @Autowired
 28     ConnectionFactory connectionFactory;
 29
 30
 31     @SuppressWarnings("unchecked")
 32     void redeliverDLQUsingJms(final String brokerName, final String from,

 33             final String to) {
 34         Connection connection = null;
 35         Session session = null;
 36 

 37         try {
 38             connection = connectionFactory.createConnection();
 39             connection.start();
 40             session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
 41             Queue dlq = session.createQueue(from);

 42             QueueBrowser browser = session.createBrowser(dlq);
 43
 44             Enumeration<Message> e = browser.getEnumeration();
 45
 46             while (e.hasMoreElements()) {

 47                 Message message = e.nextElement();
 48                 final String messageBody = ((TextMessage) message).getText();
 49                 jmsTemplate.send(to, new MessageCreator() {
 50                     @Override

 51                     public Message createMessage(final Session session)
 52                             throws JMSException {
 53                         return session.createTextMessage(messageBody);

 54                     }
 55                 });
 56             }
 57         } catch (Exception e) {
 58             throw new RuntimeException(e);

 59         } finally {
 60             try {
 61                 session.close();
 62             } catch (Exception e) {

 63             }
 64             try {
 65                 connection.close();
 66             } catch (Exception e) {

 67             }
 68         }
 69     }
 70     // . . .
 71 }
 72

The downside of above approach is that it leaves the original messages in the dead letter queue.

Copying Messages using Spring’s JmsTemplate APIs

You can effectively do the same thing with JmsTemplate provided by Spring with a bit less code, e.g.

  1    void redeliverDLQUsingJmsTemplateBrowse(final String from, final String to) {

  2         try {
  3             jmsTemplate.browse(from, new BrowserCallback() {
  4
  5                 @SuppressWarnings("unchecked")

  6                 @Override
  7                 public Object doInJms(Session session, QueueBrowser browser)
  8                         throws JMSException {
  9                     Enumeration<Message> e = browser.getEnumeration();

 10                     while (e.hasMoreElements()) {
 11                         Message message = e.nextElement();
 12                         final String messageBody = ((TextMessage) message)
 13                                 .getText();
 14                         jmsTemplate.send(to, new MessageCreator() {

 15                             @Override
 16                             public Message createMessage(final Session session)
 17                                     throws JMSException {
 18                                 return session.createTextMessage(messageBody);

 19                             }
 20                         });
 21                     }
 22                     return null;
 23                 }

 24             });
 25         } catch (Exception e) {
 26             throw new RuntimeException(e);
 27         }

 28     }
 29

Moving Messages using receive/send APIs

As I mentioned, the above approaches leave messages in the DLQ, which may not be what you want. Thus, another simple approach would be to consume messages from the dead letter queue and send it to another,e.g.

  1   public void redeliverDLQUsingJmsTemplateReceive(final String from,

  2             final String to) {
  3         try {
  4             jmsTemplate.setReceiveTimeout(100);
  5             Message message = null;

  6             while ((message = jmsTemplate.receive(from)) != null) {
  7                 final String messageBody = ((TextMessage) message).getText();
  8                 jmsTemplate.send(to, new MessageCreator() {

  9                     @Override
 10                     public Message createMessage(final Session session)
 11                             throws JMSException {

 12                         return session.createTextMessage(messageBody);
 13                     }
 14                 });
 15             }
 16         } catch (Exception e) {

 17             throw new RuntimeException(e);
 18         }
 19     }
 20

Moving Messages using ActiveMQ’s API

Finally, the best approach I found waas to use ActiveMQ’s APIs to move messags, e.g.

  1     public void redeliverDLQUsingJMX(final String brokerName, final String from,

  2             final String to) {
  3         try {
  4             final QueueViewMBean queue = brokerQuery.getQueue(from);

  5             for (int i = 0; i < 10 && queue.getQueueSize() > 0; i++) {
  6                 CompositeData[] compdatalist = queue.browse();

  7                 for (CompositeData cdata : compdatalist) {
  8                     String messageID = (String) cdata.get("JMSMessageID");
  9                     queue.moveMessageTo(messageID, to);
 10                 }

 11             }
 12         } catch (Exception e) {
 13             throw new RuntimeException(e);
 14         }

 15     }
 16

I have been using this approach and have found to be reliable for reprocessing dead letter queue, though these techniques an also be used for general queues. I am sure there are tons of alternatives including using full-fledged enterprise service bus route. Let me know if you have interesting solutions to this problem.

Posted by admin
Filed in Computing
No Comments »

PlexRBAC: an open source project for providing powerful role based security (II)

January 20th, 2010

This is continuation of my previous blog on my open source project PlexRBAC for managing role based access control. Last time, I covered REST APIs and in this blog I will cover internal domain model, RBAC APIs in Java and examples of instance or dynamic based security.

Layers

PlexRBAC consists of following layers

Business Domain Layer

This layer defines core classes that are part of the RBAC based security domain such as:

  • Domain – As described previously, the domain allows you to support multiple applications or realms.
  • Subject – The subject represents users who are defined in an application.
  • Role – A role represents job title or function.
  • Permission – A permission is composed of operation, target and an expression that is used for dynamic or instance based security.
  • SecurityError – Upon a permission failure, you can choose to store them in the database using SecurityError.

Repository Layer

This layer is responsible for accessing or storing above objects in the database. PlexRBAC uses Berkley DB for persistence and each domain is stored as a separate database, which allows you to segregate permissions and roles for distinct domains. Following are list of repositories supported by PlexRBAC:

  • DomainRepository – provides database access for Domains.
  • PermissionRepository – provides database access for Permissions.
  • SubjectRepository – provides database access for Subjects.
  • SecurityErrorRepository – provides database access for SecurityErrors.
  • RoleRepository – provides database access for Roles.
  • SecurityMappingRepository – provides APIs to map permissions with roles and to map subject with roles.
  • RepositoryFactory – provides factory methods to create above repositories.

Security Layer

This class defines PermissionManager for authorizing permissions.

Evaluation Layer

This layer proivdes evaluation engine for instance based security.

Service Layer

This layer defines REST services such as:

  • DomainService – this service provides REST APIs for accessing Domains.
  • PermissionService – this service provides REST APIs for accessing Permissions.
  • SubjectService – this service provides REST APIs for accessing Subjects.
  • RoleService – this service provides REST APIs for accessing Roles.
  • AuthenticationService – this service provides REST APIs for authenticating users.
  • AuthorizationService – this service provides REST APIs for authorizing permissions.
  • RolePermissionService – this service provides REST APIs for mapping permissions with roles.
  • SubjectRolesService – this service provides REST APIs for mapping subjects with roles.

JMX Layer

This layer defines JMX helper classes for managing services and configuration remotely.

Caching Layer

This layer provides caching security permissions to improve performance.

Metrics Layer

This layer provides performance measurement classes such as Timing class to measure method invocation benchmarks.

Utility Layer

This layer provides helper classes.

Web Layer

This layer provides filters for enforcing authentication and authorization when accessing REST APIs.

Example

Let’s use the same example that we described last time but with addition of instance based security. Let’s assume there are five roles: Teller, Customer-Service-Representative (CSR), Account, AccountingManager and LoanOfficer, where

  • A teller can modify customer deposit accounts — but only if customer and teller live in same region
  • A customer service representative can create or delete customer deposit accounts — but only if customer and teller live in same region
  • An accountant can create general ledger reports — but only if year is == current year
  • An accounting manager can modify ledger-posting rules — but only if year is == current year
  • A loan officer can create and modify loan accounts – but only if account balance is < 10000

In addition, following classes will be used to add domain specific security:

  1
  2 class User {

  3
  4     private String id;
  5     private String region;
  6 

  7     User() {
  8     }
  9
 10     public User(String id, String region) {
 11         this.id = id;

 12         this.region = region;
 13     }
 14
 15     public void setRegion(String region) {
 16         this.region = region;

 17     }
 18
 19     public String getRegion() {
 20         return region;
 21     }

 22
 23     public void setId(String id) {
 24         this.id = id;
 25     }
 26 

 27     public String getId() {
 28         return id;
 29     }
 30 }
 31 

 32 class Customer extends User {
 33
 34     public Customer(String id, String region) {
 35         super(id, region);

 36     }
 37 }
 38
 39 class Employee extends User {
 40 

 41     public Employee(String id, String region) {
 42         super(id, region);
 43     }
 44 }
 45 

 46 class Account {
 47
 48     private String id;
 49     private double balance;

 50
 51     Account() {
 52     }
 53
 54     public Account(String id, double balance) {

 55         this.id = id;
 56         this.balance = balance;
 57     }
 58
 59     /**

 60      * @return the id
 61      */
 62     public String getId() {

 63         return id;
 64     }
 65
 66     /**
 67      * @param id

 68      *            the id to set
 69      */
 70     public void setId(String id) {

 71         this.id = id;
 72     }
 73
 74     public void setBalance(double balance) {

 75         this.balance = balance;
 76     }
 77
 78     public double getBalance() {
 79         return balance;

 80     }
 81 }
 82
 83

Bootstrapping

Let’s create handle to repository-factory as:

 1
 2     private static final String TEST_DB_DIR = "test_db_dir_perms";

 3     RepositoryFactory repositoryFactory = new RepositoryFactoryImpl(TEST_DB_DIR);

And instance of permission manager as:

 1 PermissionManager permissionManager = new PermissionManagerImpl(repositoryFactory,

 2             new JavascriptEvaluator());

Creating a domain

Now, let’s create a domain for banking:

 1     private static final String BANKING = "banking";

 2     repositoryFactory.getDomainRepository().save(new Domain(BANKING, ""));

Creating Users

Next step is to create users for the domain or application so let’s define accounts for tom, cassy, ali, mike and larry, i.e.,

 1         final SubjectRepository subjectRepo = repositoryFactory

 2                 .getSubjectRepository(BANKING);
 3         Subject tom = subjectRepo.save(new Subject("tom", "pass"));
 4         Subject cassy = subjectRepo.save(new Subject("cassy", "pass"));

 5         Subject ali = subjectRepo.save(new Subject("ali", "pass"));
 6         Subject mike = subjectRepo.save(new Subject("mike", "pass"));

 7         Subject larry = subjectRepo.save(new Subject("larry", "pass"));
 8

Creating Roles

Now, we will create roles for Teller, CSR, Accountant, AccountManager and LoanManager:

  1         final RoleRepository roleRepo = repositoryFactory

  2                 .getRoleRepository(BANKING);
  3         Role employee = roleRepo.save(new Role("Employee"));
  4         Role teller = roleRepo.save(new Role("Teller", employee));

  5         Role csr = roleRepo.save(new Role("CSR", teller));
  6         Role accountant = roleRepo.save(new Role("Accountant", employee));

  7         Role accountantMgr = roleRepo.save(new Role("AccountingManager",
  8                 accountant));
  9         Role loanOfficer = roleRepo

 10                 .save(new Role("LoanOfficer", accountantMgr));
 11

Creating Permissions

We can then create new permissions and save them in the database as follows:

  1         final PermissionRepository permRepo = repositoryFactory

  2                 .getPermissionRepository(BANKING);
  3         Permission cdDeposit = permRepo.save(new Permission("(create|delete)",
  4                 "DepositAccount",

  5                 "employee.getRegion().equals(customer.getRegion())")); // 1
  6         Permission ruDeposit = permRepo.save(new Permission("(read|modify)",
  7                 "DepositAccount",

  8                 "employee.getRegion().equals(customer.getRegion())")); // 2
  9         Permission cdLoan = permRepo.save(new Permission("(create|delete)",
 10                 "LoanAccount", "account.getBalance() < 10000")); // 3

 11         Permission ruLoan = permRepo.save(new Permission("(read|modify)",
 12                 "LoanAccount", "account.getBalance() < 10000")); // 4

 13
 14         Permission rdLedger = permRepo.save(new Permission("(read|create)",
 15                 "GeneralLedger", "year == new Date().getFullYear()")); // 5

 16
 17         Permission rGlpr = permRepo
 18                 .save(new Permission("read", "GeneralLedgerPostingRules",
 19                         "year == new Date().getFullYear()")); // 6

 20
 21         Permission cmdGlpr = permRepo.save(new Permission(
 22                 "(create|modify|delete)", "GeneralLedgerPostingRules",
 23                 "year == new Date().getFullYear()")); // 7

 24

Mapping Subjects/Permissions to Roles

Now we will map subjects to roles as follows:

 1         final SecurityMappingRepository smr = repositoryFactory

 2                 .getSecurityMappingRepository(BANKING);
 3
 4         // Mapping Users to Roles
 5         smr.addRolesToSubject(tom, teller);
 6         smr.addRolesToSubject(cassy, csr);
 7         smr.addRolesToSubject(ali, accountant);

 8         smr.addRolesToSubject(mike, accountantMgr);
 9         smr.addRolesToSubject(larry, loanOfficer);
 0

Then we will map permissions to roles as follows:

 1         smr.addPermissionsToRole(teller, ruDeposit);
 2         smr.addPermissionsToRole(csr, cdDeposit);

 3         smr.addPermissionsToRole(accountant, rdLedger);
 4         smr.addPermissionsToRole(accountant, ruLoan);
 5         smr.addPermissionsToRole(accountantMgr, cdLoan);
 6         smr.addPermissionsToRole(accountantMgr, rGlpr);
 7         smr.addPermissionsToRole(loanOfficer, cmdGlpr);
 8

Authorization

Now the fun part of authorization, let’s check if user “tom” can view deposit-accounts, e.g.

  1    public static Map<String, Object> toMap(final Object... keyValues) {

  2         Map<String, Object> map = new HashMap<String, Object>();
  3         for (int i = 0; i < keyValues.length - 1; i += 2) {

  4             map.put(keyValues[i].toString(), keyValues[i + 1]);
  5         }
  6         return map;
  7     }

  8     @Test
  9     public void testReadDepositByTeller() {
 10         initDatabase();
 11         permissionManager.check(new PermissionRequest(BANKING, "tom", "read",

 12                 "DepositAccount", toMap("employee", new Employee("tom",
 13                         "west"), "customer", new Customer("zak", "west"))));

 14     }
 15
 16

Note that above test method builds a PermissionRequest that encapsulates domain, subject, operation, target and context and then calls check method of SecurityManager, which throws SecurityException if permission fails.

Then we check if tom, the teller can delete deposit-account, e.g.

 1     @Test(expected = SecurityException.class)

 2     public void testDeleteByTeller() {
 3         initDatabase();
 4         permissionManager.check(new PermissionRequest(BANKING, "tom", "delete",

 5                 "DepositAccount", toMap("employee", new Employee("tom",
 6                         "west"), "customer", new Customer("zak", "west"))));

 7     }
 8

Which would throw security exception.

Now let’s check if cassy, the CSR can delete deposit-account, e.g.

 1     @Test
 2     public void testDeleteByCsr() {

 3         initDatabase();
 4         permissionManager.check(new PermissionRequest(BANKING, "cassy",
 5                 "delete", "DepositAccount", toMap("employee",

 6                         new Employee("cassy", "west"), "customer",
 7                         new Customer("zak", "west"))));

 0

Which works as CSR have permissions for deleting deposit-account. Now, let’s check if ali, the accountant can view general-ledger, e.g.

 1    @Test
 2     public void testReadLedgerByAccountant() {

 3         initDatabase();
 4         permissionManager.check(new PermissionRequest(BANKING, "ali", "read",
 5                 "GeneralLedger", toMap("year", 2010, "account",

 6                         new Account("zak", 500))));
 7     }
 8
 9

Which works as expected. Next we check if ali can delete general-ledger:

 1     @Test(expected = SecurityException.class)

 2     public void testDeleteLedgerByAccountant() {
 3         initDatabase();
 4         permissionManager.check(new PermissionRequest(BANKING, "ali", "delete",

 5                 "GeneralLedger", toMap("year", 2010, "account",
 6                         new Account("zak", 500))));
 7     }

 8
 9

Which would fail as only account-manager can delete. Next we check if mike, the account-manager can create general-ledger, e.g.

 1     @Test
 2     public void testCreateLedgerByAccountantManager() {

 3         initDatabase();
 4         permissionManager.check(new PermissionRequest(BANKING, "mike",
 5                 "create", "GeneralLedger", toMap("year", 2010,

 6                         "account", new Account("zak", 500))));
 7     }
 8

Which works as expected. Now we check if mike can create posting-rules of general-ledger, e.g.

 1     @Test(expected = SecurityException.class)

 2     public void testPostLedgingRulesByAccountantManager() {
 3         initDatabase();
 4         permissionManager.check(new PermissionRequest(BANKING, "mike",

 5                 "create", "GeneralLedgerPostingRules", toMap("year",
 6                         2010, "account", new Account("zak", 500))));

 7     }
 8

Which fails authorization. Then we check if larry, the loan officer can create posting-rules of general-ledger, e.g.

 1     @Test
 2     public void testPostLedgingRulesByLoanManager() {

 3         initDatabase();
 4         permissionManager.check(new PermissionRequest(BANKING, "larry",
 5                 "create", "GeneralLedgerPostingRules", toMap("year",

 6                         2010, "account", new Account("zak", 500))));
 7     }
 8

Which works as expected. Now, let’s check the same permission but with different year, e.g.

 1     @Test(expected = SecurityException.class)

 2     public void testPostLedgingRulesByLoanManagerWithExceededAmount() {
 3         initDatabase();
 4         permissionManager.check(new PermissionRequest(BANKING, "larry",

 5                 "create", "GeneralLedgerPostingRules", IDUtils.toMap("year",
 6                         2011)));
 7     }
 8

Which fails as year doesn’t match.

Summary

Above examples demonstrate how PlexRBAC API can be used along with instance or dynamic based security. In next post, I will describe caching and how PlexRBAC can be integrated with J2EE and Spring security.

Posted by admin
Filed in Computing
No Comments »

PlexRBAC: an open source project for providing powerful role based security (I)

January 10th, 2010

Overview

In my last blog I described core pieces of a security system and mentioned a new open source project PlexRBAC I recently started to provide Role Based Security both as a REST service and Java library. In this post, I will go over the some of the features that are now available. This project is based on my experience with a number of home built solutions for RBAC and standard J2EE solutions. However, a key differentiator is that it adds instance based security or context based security that adds dynamic access control. The role based security consists of following components:

Domain

Though, domain is strictly not part of role based security but RBAC provides segregation of security policies by domains, where a domain can represent a security realm or an application.

Subject

The subject represents users who are defined in an application.

Role

A role represents job title or function. A subject or user belongs to one or more roles. One of key feature of PlexRBAC is that roles support inheritance where a role can have one or more roles. This helps define security policies that follow “don’t repeat yourself” or DRY.

Permission

A permission consists of two sub parts: operation and target, where operation is a “verb” that describes action and target represents “object” that is acted upon. All permissions are assigned to roles. In PlexRBAC, permissions also contains an expression which is evaluated to check dynamic security. PlexRBAC allows Javascript based expressions and provides access to runtime request parameters. Finally, PlexRBAC offers regular expressions for both operations and target, so you can define operations like “(read|write|create|delete)” or “read*”, etc.

Following diagram shows the relationship between these components:

Getting Started

PlexRBAC depends on Java 1.6+ and Maven 2.0+. You can download the project using git:

 git clone git@github.com:bhatti/PlexRBAC.git

Then you can start the REST based web service within Jetty by typing:

 mvn jetty:run-war

The service will listen on port 8080 and you can test it with curl.

Authentication

Though, PlexRBAC is not designed for authentication but it provides Basic authentication and all administration APIs are protected with the authentication. By default, it uses an account “super_admin” with password “changeme”, which you can modify with configurations. Also, as PlexRBAC supports domains to segregates security policies, subjects are also restricted to the domains where they are defined.

REST APIs

Following are APIs defined in PlexRBAC:

Domains

  • GET /api/security/domains – returns list of all domains in JSON format.
  • GET /api/security/domains/{domain-id} – returns details of given domain in JSON format.
  • PUT /api/security/domains/{domain-id} with body of domain details in JSON format.
  • DELETE /api/security/domains – deletes all domains.
  • DELETE /api/security/domains/{domain-id} – deletes domain identified by domain-id.

Subjects

  • GET /api/security/subjects/{domain-id} – returns list of all subjects in domain identified by domain-id in JSON format.
  • GET /api/security/subjects/{domain-id}/{id} – returns details of given subject identified by id in given domain.
  • PUT /api/security/subjects/{domain-id}/{id} with body of subject details in JSON format.
  • DELETE /api/security/subjects/{domain-id} – deletes all subjects in given domain.
  • DELETE /api/security/subjects/{domain-id}/{id} – deletes subject identified by id.

Roles

  • GET /api/security/roles/{domain-id} – returns list of all roles in domain identified by domain-id in JSON format.
  • GET /api/security/roles/{domain-id}/{id} – returns details of given role identified by id in given domain.
  • PUT /api/security/roles/{domain-id}/{id} with body of role details in JSON format.
  • DELETE /api/security/roles/{domain-id} – deletes all roles in given domain.
  • DELETE /api/security/roles/{domain-id}/{id} – deletes role identified by id.

Permissions

  • GET /api/security/permissions/{domain-id} – returns list of all permissions in domain identified by domain-id in JSON format.
  • GET /api/security/permissions/{domain-id}/{id} – returns details of given permission identified by id in given domain.
  • POST /api/security/permissions/{domain-id} with body of permission details in JSON format. Note that this API uses POST instead of PUT as the id will be assigned by the server.
  • DELETE /api/security/permissions/{domain-id} – deletes all permissions in given domain.
  • DELETE /api/security/permissions/{domain-id}/{id} – deletes permission identified by id.

Mapping of Roles and Permissions

  • PUT /api/security/role_perms/{domain-id}/{role-id} – adds permissions identified by permissionIds that stores list of permission-ids in JSON format. Note that permissionIds is passed as a form parameter.
  • DELETE /api/security/role_perms/{domain-id}/{role-id} – removes permissions identified by permissionIds that stores list of permission-ids in JSON format. Note that permissionIds is passed as a form parameter.

Mapping of Subjects and Roles

  • PUT /api/security/subject_roles/{domain-id}/{subject-id} – adds roles identified by rolenames that stores list of role-ids in JSON format. Note that rolenames is passed as a form parameter.
  • DELETE /api/security/subject_roles/{domain-id}/{subject-id} – removes roles identified by rolenames that stores list of role-ids in JSON format. Note that rolenames is passed as a form parameter.

Authorization

  • GET /api/security/authorize/{domain-id} – with query parameter of operation and target.

Example

Let’s start with a banking example where a bank-object can be account, general-ledger-report or ledger-posting-rules and account is further grouped into customer account or loan account, e.g.

Let’s assume there are five roles: Teller, Customer-Service-Representative (CSR), Account, AccountingManager and LoanOfficer, where

  • A teller can modify customer deposit accounts.
  • A customer service representative can create or delete customer deposit accounts.
  • An accountant can create general ledger reports.
  • An accounting manager can modify ledger-posting rules.
  • A loan officer can create and modify loan accounts.

Creating a domain

The first thing is to create a security domain for your application. As we are dealing with banking domain, let’s call our domain “banking”. 

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/domains/banking" -d '{"id":"banking"}'

It will return response:

 {"id":"banking","ownerSubjectNames":"super_admin"}

The first thing to note that we are passing user and password using Basic authentication as all accesses to administration APIs require login. Now, you can find out available domains via

 curl -v --user "super_admin:changeme" "http://localhost:8080/api/security/domains"

which would return something like:

 [{"id":"banking","ownerSubjectNames":"super_admin"},{"description":"default","id":"default","ownerSubjectNames":"super_admin"}]

Creating Users

Next step is to create users for the domain or application so let’s define accounts for tom, cassy, ali, mike and larry, i.e.,

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/subjects/banking" -d '{"id":"tom","credentials":"pass"}'
 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/subjects/banking" -d '{"id":"cassy","credentials":"pass"}'
 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/subjects/banking" -d '{"id":"ali","credentials":"pass"}'
 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/subjects/banking" -d '{"id":"mike","credentials":"pass"}'
 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/subjects/banking" -d '{"id":"larry","credentials":"pass"}'

Note that each user is identified by an id or username and credentials and in above examples usernames or subject-ids are prefixed with domain-ids, e.g. “ddefault:super_admin”.

Creating Roles

As I mentioned, a role represents job title or responsibilities and each role can have one or more parents. By default, PlexRBAC defines an “anonymous” role, which is used for users who are not logged in and all user-defined roles extend “anonymous” role.

First, we create a role for bank employee called “Employee”:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/roles/banking" -d '{"id":"Employee"}'

which returns

 {"id":"Employee","parentIds":["anonymous"]}

As you can see the “Employee” role is created with parent of “anonymous”. Next, we create “Teller” role:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/roles/banking" -d '{"id":"Teller","parentIds":["Employee"]}'

which returns:

 {"id":"Teller","parentIds":["Employee"]}

Then we create a role for customer-service-representative called “CSR” that is extended by Teller e.g.

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/roles/banking" -d '{"id":"CSR","parentIds":["Teller"]}'

which returns:

 {"id":"CSR","parentIds":["Teller"]}

Then we create a role for “Accountant”:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/roles/banking" -d '{"id":"Accountant","parentIds":["Employee"]}'

which returns:

 {"id":"Accountant","parentIds":["Employee"]}

Then we create a role for “AccountingManager”, which is extended by “Accountant”, e.g.

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/roles/banking" -d '{"id":"AccountingManager","parentIds":["Accountant"]}'

which returns:

 {"id":"AccountingManager","parentIds":["Accountant"]}

Finally, we create a role for “LoanOfficer”, e.g.

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/roles/banking" -d '{"id":"LoanOfficer","parentIds":["Employee"]}'

which returns:

 {"id":"LoanOfficer","parentIds":["Employee"]}

Creating Permissions

As described above, a permission is composed of operation, target and expression, where an operation and target can be any regular expression and expression can be any Javascript expression. However following permissions don’t define any expressions for simplicity. First, we create a permission to create or delete deposit-account, e.g.

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X POST "http://localhost:8080/api/security/permissions/banking" -d '{"operation":"(create|delete)","target":"DepositAccount","expression":""}'

which returns:

 {"expression":"","id":"1","operation":"(create|delete)","target":"DepositAccount"}

Each permission is automatically assigned a unique numeric id. Next, we create a permission to read or modify deposit-account, e.g.

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X POST "http://localhost:8080/api/security/permissions/banking" -d '{"operation":"(read|modify)","target":"DepositAccount","expression":""}'

which returns:

 {"expression":"","id":"2","operation":"(read|modify)","target":"DepositAccount"}

Then, we create a permission to create or delete loan-account

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X POST "http://localhost:8080/api/security/permissions/banking" -d '{"operation":"(create|delete)","target":"LoanAccount","expression":""}'

which returns:

 {"expression":"","id":"3","operation":"(create|delete)","target":"LoanAccount"}

Then we create a permission to read or modify loan-account, e.g.

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X POST "http://localhost:8080/api/security/permissions/banking" -d '{"operation":"(read|modify)","target":"LoanAccount","expression":""}'

which returns:

 {"expression":"","id":"4","operation":"(read|modify)","target":"LoanAccount"}

Then we create a role to view and create general-ledger, e.g.

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X POST "http://localhost:8080/api/security/permissions/banking" -d '{"operation":"(read|create)","target":"GeneralLedger","expression":""}'

which returns:

 {"expression":"","id":"5","operation":"(read|create)","target":"GeneralLedger"}

Finally, we create a permission for modifying posting rules of general-ledger, e.g.

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X POST "http://localhost:8080/api/security/permissions/banking" -d '{"operation":"(read|create|modify|delete)","target":"GeneralLedgerPostingRules","expression":""}'

which returns:

 {"expression":"","id":"6","operation":"(read|create|modify|delete)","target":"GeneralLedgerPostingRules"}

Mapping Permissions to Roles

Next task is to map permissions to roles. First we assign permission to view or modify customer deposit accounts to Teller role:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/role_perms/banking/Teller" -d 'permissionIds=["2"]'

which returns all permission-ids for given role, e.g.

 ["2"]

Then we assign permission to view, create, modify or delete customer deposit accounts to CSR (as CSR extends Teller it will automatically will get all permissions of Teller):

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/role_perms/banking/CSR" -d 'permissionIds=["1"]'

Then we assign permissions to create general ledger to Accountant:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/role_perms/banking/Accountant" -d 'permissionIds=["5"]'

Then we assign permission to modify ledger-posting rules to AccountingManager:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/role_perms/banking/AccountingManager" -d 'permissionIds=["6"]'

Mapping Users to Roles

A role is associated with one or more permissions and each user is assigned one or more role. First, we assign subject “tom” to Teller role:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/subject_roles/banking/tom" -d 'rolenames=["Teller"]'

which returns list of all roles for given subject or user, e.g.

 ["Teller"]

Then we assign subject “cassy” to CSR role:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/subject_roles/banking/cassy" -d 'rolenames=["CSR"]'

Next we assign subject “ali” to role of Accountant:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/subject_roles/banking/ali" -d 'rolenames=["Accountant"]'

Then we assign role AccountingManager to “mike”:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/subject_roles/banking/mike" -d 'rolenames=["AccountingManager"]'

Finally we assign subject “larry” to LoanOfficer role:

 curl -H "Content-Type: application/json" --user "default:super_admin:changeme" -X PUT "http://localhost:8080/api/security/subject_roles/banking/larry" -d 'rolenames=["LoanOfficer"]'

Authorization

Now we are ready to validate authorization based on above security policies. For example, let’s check if user “tom” can view deposit-accounts, e.g.

 curl -v --user "banking:tom:pass" "http://localhost:8080/api/authorize/banking?operation=read&target=DepositAccount"

On successful authorization, the API returns 200 http responose-code and on failure it returns 401 http response-code, e.g.

 < HTTP/1.1 200 OK

Then we check if tom, the teller can delete deposit-account, e.g.

 curl -v --user "banking:tom:pass" "http://localhost:8080/api/authorize/banking?operation=delete&target=DepositAccount"

which returns http-response-code 401, e.g.

 < HTTP/1.1 401 Unauthorized

Then we create if cassy, the CSR can delete deposit-account, e.g.

 curl -v --user "banking:cassy:pass" "http://localhost:8080/api/authorize/banking?operation=delete&target=DepositAccount"

which returns:

 < HTTP/1.1 200 OK

Then we check if ali, the accountant can view general-ledger, e.g.

 curl -v --user "banking:ali:pass" "http://localhost:8080/api/authorize/banking?operation=read&target=GeneralLedger"

which returns:

 < HTTP/1.1 200 OK

Next we check if mike, the accounting-manager can create general-ledger, e.g.

 curl -v --user "banking:mike:pass" "http://localhost:8080/api/authorize/banking?operation=create&target=GeneralLedger"

which returns:

 < HTTP/1.1 200 OK

Then we check if larry, the loan officer can create posting-rules of general-ledger, e.g.

 curl -v --user "banking:mike:pass" "http://localhost:8080/api/authorize/banking?operation=create&target=GeneralLedgerPostingRules"

which returns:

 < HTTP/1.1 200 OK

Next, ali tries to create posting rules via

 curl -v --user "banking:ali:pass" "http://localhost:8080/api/authorize/banking?operation=create&target=GeneralLedgerPostingRules"

which is denied:

 < HTTP/1.1 401 Unauthorized

Summary

Above examples demonstrate how PlexRBAC can be used to define and enforce flexible security policies. In next post, I will describe instance based security, regular expressions and Java APIs for PlexRBAC.

Posted by admin
Filed in Computing
No Comments »
« Previous Entries