2012-12-12

Dokuwiki + Slideshow + Syntaxhighlighter 3

Wonder why I can't Google anything like this. I was using Dokuwiki to make netty slides in Linux (as Libreoffice Impress is slow and easy to crash). Everything was fine, just that deck.js doesn't support Syntaxhighlighter 3 to the extent of Dokuwiki. And there is a final strike: DIV size varies according to the content, the fewer text in it, the larger font-size is. Ugly. So I downloaded S5 Reloaded plugin for Dokuwiki and tweaked quite a bit to make it work with Syntaxhighlighter. First, choose a theme for "presentation" only (hence the theme in the wiki is different from that on slides). I chose shThemeDark.css. Vim your syntaxhighlighter3/sxh3/styles/shThemeRDark.css, patch one line:
.syntaxhighlighter {
  background-color: #1b2426 !important;
  font-size: 14px !important;  /* ADD THIS */
}
Or it inherits font-size: 19px from S5. Then, patch s5reloaded/renderer.php.
<link rel="stylesheet" type="text/css" href="/dokuwiki/lib/plugins/syntaxhighlighter3/sxh3/styles/shCore.css"/>
<link rel="stylesheet" type="text/css" href="/dokuwiki/lib/plugins/syntaxhighlighter3/sxh3/styles/shThemeRDark.css"/>
<script type="text/javascript" charset="utf-8" src="/dokuwiki/lib/plugins/syntaxhighlighter3/sxh3/scripts/shCore.js"></script>
<script type="text/javascript" charset="utf-8" src="/dokuwiki/lib/plugins/syntaxhighlighter3/sxh3/scripts/shAutoloader.js"></script>
And put SyntaxHighlighter.autoloader in document_end(). That's all. Enjoy!

2012-11-24

ffmpeg/avconv Parameters for Kindle Fire

Just taking a note for myself:
# For libx264
apt-get install ffmpeg libavcodec-extra-53
# For input < 1024x600
avconv -i INPUT -acodec aac -ab 96k -vcodec libx264 -f mp4 -crf 22 -strict experimental OUTPUT.mp4
# For input > 1024x600
ffmpeg -i INPUT -acodec aac -ab 96k -vcodec libx264 -f mp4 -crf 22 -s 1024x600 -strict experimental OUTPUT
Copied from ConversionMatrix, only that -vpre slow is no longer supported.

2012-08-06

clojure-hbase with filter

The project I'm working on got switched from MySQL to HBase in order to react against some massive prediction of data. Thank David Santiago for his brilliant clojure-hbase project, I can access HBase in Clojure. Here are some tips if you have to work in some unfriendly enterprise environment like I am. After all, Clojure is our rejoice! :)
  • Tip #1. Customised HBase CLASSPATH
Forget about resources/* in your Leiningen project. Just lein jar and make a run.sh:
#!/bin/bash
CP=$(lein classpath)
java -cp "/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/etc/hbase/conf:/usr/lib/hbase/hbase.jar:/usr/lib/hbase/lib/*:./target/*:$CP" myproject.core $*
  • Tip #2. Filter
You may use filter in hb/get and hb/scan. Just follow this, and you get all rows "U-*" whose columns begin with "M-d5".
(defn prefix-filter
  [prefix]
  (ColumnPrefixFilter. (Bytes/toBytes prefix)))

(defn test-scanner
  [^HTable mr]
  (hb/with-scanner [results (hb/scan mr
                                     :start-row "U-"
                                     :stop-row "U."
                                     :filter (prefix-filter "M-d5")
                                     )]
                   (doall (map #(println %)
                               (seq results)))))

2012-04-26

新手的 Neo4j: node_auto_index

跟著 Cypher Cookbook玩耍的時候,遇上了一個怪問題:
ljm:~/neo4j$ ./neo4j-community-1.7/bin/neo4j-shell 
NOTE: Remote Neo4j graph database service 'shell' at port 1337
Welcome to the Neo4j Shell! Enter 'help' for a list of commands

neo4j-sh (0)$ START a=node:node_auto_index(name="Thomas Anderson") RETURN a
MissingIndexException: Index `node_auto_indedex` does not exist
啥? 我 Google 了一下,發現要把 neo4j.properties 裡面的 node_auto_indexing 打開,所以:
node_auto_indexing=true
node_keys_indexable=name,age
relationship_auto_indexing=true
relationship_keys_indexable=IS_ROOT_OF,KNOWS,CODED_BY
咦? 還是沒有用? 再 Google 了一下,發現打開後 **新增的 nodes** 才有 index,因為它是 lazy index。 把 nodes 砍掉重練,於是可以用 Cypher 了... 試試看,很好玩!
neo4j-sh (0)$ START a=node:node_auto_index(name="Cypher"),
b=node:node_auto_index(name="Thomas Anderson")
MATCH p=shortestPath(a-[r*..10]-b)
RETURN p     
+--------------------------------------------+
| p                                          |
+--------------------------------------------+
| (18)<--[KNOWS,22]--(16)<--[KNOWS,19]--(15) |
+--------------------------------------------+
1 row, 1 ms
要從 neo4jrestclient 來 query 也可以:
>>> c = gdb.extensions.CypherPlugin
>>> r = c.execute_query("START a=node:node_auto_index(name='Trinity') MATCH a-->b RETURN a,b")
>>> [d['self'] for d in r['data'][0]]
[u'http://localhost:7474/db/data/node/30', u'http://localhost:7474/db/data/node/32']
>>> [d['data'] for d in r['data'][1]]
[{u'name': u'Trinity'}, {u'age': 29, u'name': u'Thomas Anderson'}]
>>> [d['data'] for d in r['data'][0]]
[{u'name': u'Trinity'}, {u'lastname': u'Reagan', u'name': u'Cypher'}]
再列一些參考資料:

Histogram in Clojure

學到 (rand-int) 的時候,想說應該驗證一下亂數產生器的分佈夠不夠均勻,可是不知道怎麼寫 histogram. 下面的程式參考了 Along Came Betty:
(def randmap (for [x (range 100)] (rand-int 10)))
==> #'user/randmap
(reduce (fn [m w] (assoc m w (+ 1 (m w 0)))) {} randmap)
==> {0 9, 1 7, 2 11, 3 14, 4 9, 5 7, 6 11, 7 11, 8 9, 9 12}
覺得 0 1 2 3 ... 礙眼?
(vals (reduce (fn [m w] (assoc m w (+ 1 (m w 0)))) {} randmap))
==> (9 7 11 14 9 7 11 11 9 12)
嗯,看來這個亂數產生器不夠靠譜 XD

2012-04-25

新手的 Neo4j: plugins

neo4j-server-examples 下載常用的 plugin ,然後重新啟動 neo4j-service。 好了,可以接上 這一頁 的範例了...
>>> from neo4jrestclient.client import GraphDatabase
>>> gdb = GraphDatabase("http://localhost:7474/db/data/")
>>> gdb.extensions
{u'CypherPlugin': <Neo4j ExtensionModule: [u'execute_query']>, u'GremlinPlugin': <Neo4j ExtensionModule: [u'execute_script']>, u'GetAll': <Neo4j ExtensionModule: [u'get_all_nodes', u'getAllRelationships']>}


>>> gdb.node[5].properties
{u'name': u'Trinity'}
>>> gdb.node[9].properties
{u'name': u'The Architect'}
>>> trinity = gdb.node[5]
>>> # 找出從 Trinity 到 The Architect 的最短路徑
>>> p = trinity.extensions.ShortestPath.shortestPath(gdb.node[9])
>>> p
[{u'relationships': [u'http://localhost:7474/db/data/relationship/8', u'http://localhost:7474/db/data/relationship/9', u'http://localhost:7474/db/data/relationship/10'], u'start': u'http://localhost:7474/db/data/node/5', u'nodes': [u'http://localhost:7474/db/data/node/5', u'http://localhost:7474/db/data/node/7', u'http://localhost:7474/db/data/node/8', u'http://localhost:7474/db/data/node/9'], u'end': u'http://localhost:7474/db/data/node/9', u'length': 3}]
>>> print [gdb.node[n].properties['name'] for n in p[0]['nodes']]
[u'Trinity', u'Cypher', u'Agent Smith', u'The Architect']

2012-04-24

新手的 Neo4j + Python

因為我在公司使用 Ubuntu 11.10, 參考這一頁安裝時,需要留意一下... 不需要安裝 Neo4j community 1.7。 如果要安裝 neo4j-embedded 的話:
  1. 直接 sudo apt-get install python-jpype ,不用下載了。
  2. 如果一定要下載來自己編譯,可以用 sudo JAVA_HOME=/usr/lib/jvm/java-6-sun python setup.py install
  3. sudo apt-get install python-pip
  4. sudo pip install neo4j-embedded
就可以開始玩了... 網路上有好多 demo, 寫得好複雜的樣子,晚一點再來看有沒有簡單一點的。 啊。搞錯了。 用 neo4j-embedded 雖然沒什麼問題,但是公司的應用必須有 standalone server,而且把服務跑起來,才能用 browser,所以要安裝 neo4j-community:
  1. sudo ./neo4j-community-1.7/bin/neo4j install 安裝服務
  2. sudo -u neo4j /etc/init.d/neo4j-service start 把Neo4j打開
  3. 用瀏覽器看 http://localhost:7474/webadmin/
Python 可以用的 RESTful client 有兩個: neo4jrestclientpy2neo。 公司的 VM 上要安裝新軟體很痛苦,所以我本來應該裝 py2neo,可是因為 neo4jrestclient 包裝得和 neo4j-embedded 很像,所以下面的例子用的是 neo4jrestclient:
# 有名的 The Matrix 的例子
from neo4jrestclient.client import GraphDatabase
db = GraphDatabase("http://localhost:7474/db/data/")

thomas = db.node(name = 'Thomas Anderson', age = 29)
trinity = db.node(name = 'Trinity')
morpheus = db.node(name = 'Morpheus', rank = 'Captain', occupation = 'total badass')
cypher = db.node(name = 'Cypher', lastname = 'Reagan')
agentSmith = db.node(name = 'Agent Smith', version = '1.0b', language = 'C++')
architect = db.node(name = 'The Architect')
thomas.KNOWS(trinity)
trinity.LOVES(thomas, since = 'meeting the oracle')
thomas.KNOWS(morpheus)
trinity.KNOWS(cypher, since = 'a year before the movie', cooperatesOn = 'The Nebuchadnezzar')
cypher.KNOWS(agentSmith, disclosure = 'secret')
agentSmith.CODEDBY(architect)
搞定後,就可以用 python 命令列來玩耍:
>>> from neo4jrestclient.client import GraphDatabase
>>> gdb = GraphDatabase("http://localhost:7474/db/data/")
>>> n = gdb.node[5] # 假設 Trinity 是 node[5] 可以去 browser 看 :)
>>> n.relationships.all()[:]
[<Neo4j Relationship: http://localhost:7474/db/data/relationship/8>, <Neo4j Relationship: http://localhost:7474/db/data/relationship/5>, <Neo4j Relationship: http://localhost:7474/db/data/relationship/6>]
>>> n.relationships.all()[0].type
'KNOWS'
>>> n.relationships.all()[0].start
<Neo4j Node: http://localhost:7474/db/data/node/5>
>>> n.relationships.all()[0].end
<Neo4j Node: http://localhost:7474/db/data/node/7>
>>> n.relationships.all()[0].properties
{u'since': u'a year before the movie', u'cooperatesOn': u'The Nebuchadnezzar'}
然後是 traversals:
>>> from neo4jrestclient import client
>>> n.traverse(types=[client.All.KNOWS])[:]
[<Neo4j Node: http://localhost:7474/db/data/node/7>, <Neo4j Node: http://localhost:7474/db/data/node/4>]
>>> n.traverse(types=[client.Outgoing.KNOWS])[:]
[<Neo4j Node: http://localhost:7474/db/data/node/7>]
>>> n.traverse(types=[client.Incoming.KNOWS])[:]
[<Neo4j Node: http://localhost:7474/db/data/node/4>]
比方說想找 Trinity KNOWS 的人 KNOWS 的... (我為了舉例多加了一些關係)
# Ruby 寫起來好漂亮 T_T ==> me.outoging(:frienship).outgoing(:datingship)
>>> [n.traverse(types=[Outgoing.KNOWS])[:] for n in trinity.traverse(types=[Outgoing.KNOWS])[:]]
[[<Neo4j Node: http://127.0.0.1:7474/db/data/node/8>], [<Neo4j Node: http://127.0.0.1:7474/db/data/node/5>, <Neo4j Node: http://127.0.0.1:7474/db/data/node/4>]]
# 也可以接上 neo4j 的範例...
>>> gdb.traversal().relationships('KNOWS').traverse(trinity).nodes[:]
[<Neo4j Node: http://127.0.0.1:7474/db/data/node/7>, <Neo4j Node: http://127.0.0.1:7474/db/data/node/6>, <Neo4j Node: http://127.0.0.1:7474/db/data/node/4>]
# Incoming 和 Outgoing 直接偷偷用關鍵字...
>>> gdb.traversal().relationships('KNOWS', 'out').traverse(trinity).nodes[:]
[<Neo4j Node: http://127.0.0.1:7474/db/data/node/7>, <Neo4j Node: http://127.0.0.1:7474/db/data/node/6>]
>>> gdb.traversal().relationships('KNOWS', 'in').traverse(trinity).nodes[:]
[<Neo4j Node: http://127.0.0.1:7474/db/data/node/4>]

2012-04-19

Migrate "Simple Web App" to Clojure 1.3.0

Another trial to migrate Mark McGranaghan's awesome example of Developing and Deploying a Simple Clojure Web Application to Clojure 1.3.0. First, project.clj: remove clojure-contrib and refresh the revisions...
defproject adder "0.0.1"
  :description "Add two numbers."
  :dependencies
  [[org.clojure/clojure "1.3.0"]
   [ring/ring-core "1.1.0-RC1"]
   [ring/ring-devel "1.1.0-RC1"]
   [ring/ring-jetty-adapter "1.1.0-RC1"]
   [compojure "1.0.2"]
   [hiccup "1.0.0-RC2"]]
  :dev-dependencies
  [[lein-run "1.0.0-SNAPSHOT"]])
hiccup.page-helper became hiccup.page, and we have to add ring.middleware.params after Compojure 0.6.0!!! (Really painful to find out why POST isn't working anymore.) In core.clj:
(:use [compojure.core]
      [hiccup.core]
      [hiccup.page]
      [ring.middleware.reload]
      [ring.middleware.stacktrace]
      [ring.util.response]
      [ring.middleware.file]
      [ring.middleware.file-info]
      [ring.middleware.params :only [wrap-params]])

; and for app ...
(def app
  (-> #'handler
      (wrap-params handler)
      (wrap-utf)
      (wrap-file "public")
      (wrap-file-info)
      (wrap-request-logging)
      (wrap-if development? wrap-reload '[adder.middleware adder.core])
      (wrap-bounce-favicon)
      (wrap-if production?  wrap-failsafe)
      (wrap-if development? wrap-stacktrace)))
Others are just like they were in Mark's brilliant tutorial. The source code is available here.

Migrate "REST" to Clojure 1.3.0 with Slingshot

I'm really inspired by Mark McGranaghan's Building REST APIs for Clojure Web Applications. It uses Clojure 1.2.0, but I'm really fancy at new things and wanted to port the code to Clojure 1.3.0. I learned that clojure.contrib is no longer in use and clojure.contrib.condition is now replaced by Steve Gilardi's Slingshot. So I hacked a bit ... I'm not sure if my codes are good enough. Comments are more than welcome. Changes in web.clj:
  (:use [compojure.core]
        [ring.middleware.json-params]
        [slingshot.slingshot :only [try+]])
; ...
(defn wrap-error-handling [handler]
  (fn [req]
      (try+
       (or (handler req)
           (json-response {"error" "resource not found"} 404))
       (catch JsonParseException e
              (json-response {"error" "malformed json"} 400))
       (catch [:type :bad-req] {:keys [reason message]}
              (json-response {"error" message} (error-codes reason))))))
Changes in elem.clj
  (:use [slingshot.slingshot :only [throw+]])
; ...
(defn get [id]
  (or (@elems id)
      (throw+ {:type :bad-req :reason :not-found
              :message (format "elem '%s' not found" id)})))

(defn put [id attrs]
  (if (empty? attrs)
      (throw+ {:type :bad-req :reason :invalid
              :message "attrs are empty"})
; ...
And it passes test.sh: (Sorry! I don't know how to write lein test cases.)
{}
{"error":"elem '1' not found"}
{"group":"bar"}
{"group":"biz","tag":"bat"}
{"group":"bar"}
{"2":{"group":"biz","tag":"bat"}}
{"error":"elem '3' not found"}
{"error":"resource not found"}
{"error":"malformed json"}400
{"error":"elem '3' not found"}404
{"error":"attrs are empty"}400
You may download the example here.

2012-04-14

Clojure

大部份資料都 Google 得到,這裡只是做個筆記。 下載 1.3.0 版後,直接解開來就可以用了。當然如果安裝 lein 會更優。lein 會把所有的東西都放在自己的目錄裡,所以在這家公司也沒問題。
C:\download\clojure-1.3.0>java -jar clojure-1.3.0.jar
Clojure 1.3.0
user=> (println "Hello World")
Hello World
nil
user=> (javax.swing.JOptionPane/showMessageDialog nil "Hello World")
nil
user=>
資料: 筆記:
; System.out.println()
(println "Hello World")
; Swing
(javax.swing.JOptionPane/showMessageDialog nil "Hello World")
;
; Name space
;
(def v1 10)     ; user/v1
(in-ns 'myapp)
(def v1 100)    ; myapp/v1
(in-ns 'user)
v1              ; = 10
myapp/v1        ; = 100
;
; Polymorphism
;
(defn force-it
    "The first function a young jedi needs"
    ([]
        (str "Use the force, Luke"))
    ([jedi]
        (str "Use the force, " jedi)))
;
; Lambda
;
(map (fn [x] (* 2 x)) v)  ; v = (2 4 6 8)
(map #(* 2 %) v)          ; #() = lambda
;
;
;
;
;
;
(take 10 (cycle [1 2 3 4]))
; ==> (1 2 3 4 1 2 3 4 1 2)
(interleave [:a :b :c :d :e] [1 2 3 4 5])
; ==> (:a 1 :b 2 :c 3 :d 4 :e 5)
(partition 3 [1 2 3 4 5 6 7 8 9 10])
; ==> ((1 2 3) (4 5 6) (7 8 9))
(map vector [:a :b :c :d :e] [1 2 3 4 5])
; ==> ([:a 1] [:b 2] [:c 3] [:d 4] [:e 5])
(apply str (interpose \, "asdf"))
; ==> "a,s,d,f"
(interpose \, "asdf")
; ==> (\a \, \s \, \d \, \f)
(System/getProperties)
; ==> #<Properties {java.runtime.name=Diablo Java(TM) SE Runtime Environment, sun.boot.library.path=/usr/local/diablo-jdk1.6.0/jre/lib/amd64, ...
(first (System/getProperties))
; ==> #<Entry java.runtime.name=Diablo Java(TM) SE Runtime Environment>
(.getBytes "Hello")
; ==> #<byte[] [B@5eea3cdf>
(first (.getBytes "Hello"))
; ==> 72
(re-seq #"\w+" "the quick brown fox")
; ==> ("the" "quick" "brown" "fox")
(sort (re-seq #"\w+" "the quick brown fox"))
; ==> ("brown" "fox" "quick" "the")
(java.io.File. ".")
; ==> #<File .>
(take 5 (file-seq (java.io.File. ".")))
; ==> (#<File .> #<File ./.cshrc> #<File ./.login> #<File ./.mail_aliases> #<File ./.login_conf>)
(count (file-seq (java.io.File. ".")))
; ==> 16760
(with-open [rdr (reader "README")] (count (line-seq rdr)))
; ==> CompilerException java.lang.RuntimeException: Unable to resolve symbol: reader in this context, compiling:(NO_SOURCE_PATH:5)
(use '[clojure.contrib.repl-utils :only (show)])
; ==< nil
(show java.util.Collections "sort")
; ==< ===  public java.util.Collections  ===
; ==< [11] static checkedSortedMap : SortedMap (SortedMap,Class,Class)
; ==< [12] static checkedSortedSet : SortedSet (SortedSet,Class)
; ==< [40] static sort : void (List)
; ==< [41] static sort : void (List,Comparator)
; ==< [47] static synchronizedSortedMap : SortedMap (SortedMap)
; ==< [48] static synchronizedSortedSet : SortedSet (SortedSet)
; ==< [53] static unmodifiableSortedMap : SortedMap (SortedMap)
; ==< [54] static unmodifiableSortedSet : SortedSet (SortedSet)
(slurp "readme.txt")
; ==< 讀出 readme.txt 的內容
(use '[clojure.contrib.duck-streams :only (read-lines)])
; ==< nil
(doseq [line (read-lines "readme.txt")] (println line))
; ==< 印出 readme.txt 的內容
(java.io.File. "readme.txt")
; ==< #>File readme.txt<
(reader (java.io.File. "readme.txt"))
; ==< #>BufferedReader java.io.BufferedReader@42db681c<
(line-seq (reader (java.io.File. "readme.txt")))
; ==< 印出 readme.txt 的內容
(with-open [rdr (java.io.BufferedReader. (java.io.FileReader. "readme.txt"))] (doseq [line (line-seq rdr)] (println line)))
; ==< 印出 readme.txt 的內容
(doc line-seq)
; ==< java.io.BufferedReader 的 lazy-seq!
(def book {:title "1Q84" :author "村上春樹"})
; ==< #'user/book
(assoc book :publisher "新潮社")
; ==< {:publisher "新潮社", :title "1Q84", :author "村上春樹"}
book
; ==< {:title "1Q84", :author "村上春樹"}

Jython, Jyson / simplejson, SQLite

因為公司的開發用 VM 限制重重,前輩說他們覺得最優的方式,就是在自己的目錄下放一堆 JAR ... well, 還好 Python 也做得到。
  • Jython - 直接安裝在自己的目錄下,沒問題。
  • JSON - 使用 Jyson. 他們的首頁壞掉了,可以在 這裡下載 ,或是使用 simplejson
  • SQLite3 - 用 JDBC 沒問題,我用 Xerial 版而不是 Zentus 版。 這裡有範例, 測試過了可以用。
  • HBase - 還沒試,晚一點更新上來
有這堆限制、卻又很不想學 Java ,或許該來看看 Clojure ...

2012-04-11

Ubuntu 11.10 的遠端桌面; JSON for Python 2.4

官網上寫了很多方法,我試過 vino, XDMCP, x11vnc, 最後以 x11vnc 勝出。 直接安裝就好了,按這裡的寫法,如果你已經登錄了,就用 ssh 開一個視窗:
x11vnc -display :0
如果你還沒登錄,一樣用 ssh 開一個視窗:
sudo x11vnc -auth /var/run/lightdm/root/:0 - display :0
建議使用 x11vnc -storepasswd 設定密碼,不然就要記得擋掉 port 5900 的外部存取,然後開 SSH Tunnel 。 客戶端我試用了 remmina, 沒有想像的方便、好用。 新的公司有些文化因素,使得開發人員沒辦法使用比較新的程式語言,安裝模組也會遇到很大的阻力。隨手記一下在 Python 2.4 / 2.6 都能用 JSON 模組的方法: (從 這裡 看到的)
try:
    import json
except ImportError:
    import simplejson as json
補充: 如果沒辦法編譯 simplejson 的 C 加速模組也沒關係,改一下 setup.py 不要讓它 build, 還是可以用單純用 Python 寫的模組。 後記:Python 2.6 的 json 模組實在太慢了... 預設使用 simplejson 比較快。

2012-04-05

SBCL for CentOS 6.2 x64

單純是想在 CentOS 6.2 x86_64 上安裝 SBCL 時遇上了一些問題。 SBCL 下載頁 裡的 AMD64 不能直接用,它需要 /lib64 裡的 GLIBC 2.14,但 CentOS 6.2 附的是 2.12。 解法如下:
  1. 下載 x86 版的 binary ,安裝。這版也可以直接用。
  2. 下載原始碼,編譯。
自己編譯的 AMD64 版在 make test 時會有一個 error (和多緒有關的),不知道會有什麼影響...

2012-03-28

給懂 C++ 的人的 Java 入門

最近因為轉換跑道,需要學 Java ,可是市面上的書大多是從頭教起的。 Java for C++ Programmers 是給已經懂 C++ 和基本的資料結構的人用的線上教學。值得一看!