2013-08-12

Not-so-standard HBase with Filters in Clojure

A more-or-less customised version of Hadoop/HBase is used in where I work. One day, I really wanted to use HBase filters and began to code like everyone else:
(:import [org.apache.hadoop.hbase.util Bytes]
         [org.apache.hadoop.hbase.client HTable]
         [org.apache.hadoop.hbase.filter
          RowFilter
          CompareFilter$CompareOp
          RegexStringComparator])
; ...
(defn daily-row-filter
  []
  (RowFilter. (CompareFilter$CompareOp/EQUAL
                (RegexStringComparator. "^[0-9a-fA-F-]{36}:[0-9]{8}$"))))
; ...
(hb/with-scanner [results (hb/scan ht :filter (daily-row-filter)
                            ; ...
)])
The programme happily compiled on my laptop, but I always got a class not found in CompareFilter$CompareOp/EQUAL Frustrated, I checked my colleague's code in Java and found what he wrote was:
import org.apache.hadoop.hbase.filter.CompareOp;
// ...
new RowFilter(CompareOp.EQUAL, new RegexStringComparator(...));
WHAT? It turned out that our HBase deviated from the standard. Here's the solution. In project.clj, add this not-so-standard hbase.jar:
            :resource-paths [ "/usr/lib/hbase/hbase.jar" ]
This way, you can successfully lein jar. The downside is that you cannot compile it in a standard environment any more. In core.clj, do this:
(:import
            [org.apache.hadoop.hbase.filter
            RowFilter
            CompareOp
            RegexStringComparator])

(defn daily-row-filter
  []
  (RowFilter. CompareOp/EQUAL
              (RegexStringComparator. "^[0-9a-fA-F-]{36}:[0-9]{8}$")))

  (hb/with-scanner [results (hb/scan ht
                                     :filter (daily-row-filter) ] ; ...
  )
Hope I can help someone, or at least I'm helping myself of the future.

沒有留言: