2014-12-21

mt7620n 玩玩看之一

從刀哥那搞來了一塊 mt7620n (他的blog) ,玩玩看。這塊板子原本的用途是 Wifi-HDD 。
和他一樣,在 Ubuntu 14.04.1 上採用 debootstrap 先裝個 chroot 環境,再開發,順便偷用他的 chroot.sh
用 debootstrap 或 VirtualBox 裝 32-bit VM 都可以。事實上 32-bit VM 只要 1GB 記憶體 2*CPU 就編得很順了。
S3:~/mtk7620$ sudo debootstrap  --arch=i386  wheezy mt7620_wheezy http://ftp.tw.debian.org/debian
S3:~/mtk7620$ wget https://github.com/hbtsai/devutils/raw/master/chroot.sh
S3:~/mtk7620$ chmod +x chroot.sh
S3:~/mtk7620$ ./chroot.sh mt7620_wheezy
進去後先把 /root/.bashrc改一下,加進 PS1='\u@mtk7620:\W\$ ' 再安裝必要的套件:
root@mtk7620:~# apt-get install vim build-essential git screen libncurses5-dev locales bison flex gawk zlib1g-dev liblzma-dev
root@mtk7620:~# cd /bin
root@mtk7620:~# ln -sf bash sh
把 locale 做好
修改 /etc/locale.gen 打開 en_US.UTF-8
root@mtk7620:source# apt-get install locales
root@mtk7620:source# locale-gen
export LC_ALL=en_US.UTF-8

嗯,對,我比較喜歡 dash ... 可是不改過不了 XD。
接著就用 SDK 裡面附的 QuickStart Guide 來處理。因為文件是 confidential 的關係,這邊只能講個密訣 XDDD

  • 裝 32 位元的就好, 64bit gcc 不用裝
  • 跳過 uboot 那段
  • 直接進 source 做 make menuconfig 照手冊做
  • 不能編譯 pcitools-3.0.0 的問題,在比較新的 git repo 裡的應該修好了
make dep
make
mkdir romfs

2013-11-05

Delete all versions of a column with Clojure-HBase

To delete all versions of a given column, we call Delete.deleteColumns(family, quantifier) in Java. However, it is not yet implemented in Clojure-HBase. I have submitted an issue and a pull-request. Before the pull-request is accepted, you can delete a row in the right way like this:
(hb/with-table [ht (hb/table TABLE)]
  (io!
    (.delete ht (.deleteColumns (new Delete (tobyte ROWKEY))
                                (tobyte COLUMNFAMILY) (tobyte COLUMNQUANTIFIER)))))

DON'T DO

I did something stupid like calling:
(hb/with-table [ht (hb/table TABLE)]
  (hb/delete ROWKEY :with-timestamp-before
             3000000000   ; DON'T TO THIS
             [[:column CF CQ]]))
In fact, HBase inserts a DELETE mark with the timestamp, meaning that you can no longer put a new value before it is expired.

2013-09-11

node.js + Express 3.x + session + cookie

It wasted me more than 6 hours to Google through the topic without reaching an answer. As I tried to write some examples in book "Node.js: Up and Running" in LiveScript, I bumped into this weird behaviour. Chrome v26 / Linux did not store my sessionID in the cookie, or more exactly, it stored but I couldn't retrieve it as I reload the page, resulting in that session is always regenerated. Interestingly, Firefox works properly, and so do Chrome v22/Linux and Chrome v29/Windows. I tried to write a small piece in both PHP and LiveScript. Cookie works. So the problem must be Express 3.x.
# For finding the root cause of 7-24 on Chrome 26

require! express

app = express!
app.use express.cookieParser!

app.get '/visit', (req, res) !->
        visit = parseInt req.cookies.visit or 0
        res.cookie 'visit', visit + 1
        res.send 'Visit #' + visit

app.get '/clear', (req, res) !->
        res.clearCookie 'visit'
        res.send 'Cookie cleared.'

app.listen 8001
I noticed that from Express 2.x to 3.x, the connect / anonymous function has been changed from (req, res) to (req, res, next). Nice bet, the answer is 42. Here's the code that works:
app.configure !->
        app.use express.cookieParser 'sign'
        app.use express.session {
                secret: 'secretKey'
                store: store
        }
        app.use (req, res, next) !->
                sess = req.session
                # console.log "store = "
                # console.log store
                # console.log "sess.email = " + sess.email
                res.render 'socket.jade', {email: sess.email || 'noSessMail'}
                console.log "SessionID in Express = " + req.sessionID
                next and next!   # NOTICE THIS LINE
I don't know how to use express.cookieSession(), though. Not many examples are there. You can find the Git repository to my practice here.

2013-08-12

Not-so-standard HBase with Filters in Clojure

A more-or-less customised version of Hadoop/HBase is used in where I work. One day, I really wanted to use HBase filters and began to code like everyone else:
(:import [org.apache.hadoop.hbase.util Bytes]
         [org.apache.hadoop.hbase.client HTable]
         [org.apache.hadoop.hbase.filter
          RowFilter
          CompareFilter$CompareOp
          RegexStringComparator])
; ...
(defn daily-row-filter
  []
  (RowFilter. (CompareFilter$CompareOp/EQUAL
                (RegexStringComparator. "^[0-9a-fA-F-]{36}:[0-9]{8}$"))))
; ...
(hb/with-scanner [results (hb/scan ht :filter (daily-row-filter)
                            ; ...
)])
The programme happily compiled on my laptop, but I always got a class not found in CompareFilter$CompareOp/EQUAL Frustrated, I checked my colleague's code in Java and found what he wrote was:
import org.apache.hadoop.hbase.filter.CompareOp;
// ...
new RowFilter(CompareOp.EQUAL, new RegexStringComparator(...));
WHAT? It turned out that our HBase deviated from the standard. Here's the solution. In project.clj, add this not-so-standard hbase.jar:
            :resource-paths [ "/usr/lib/hbase/hbase.jar" ]
This way, you can successfully lein jar. The downside is that you cannot compile it in a standard environment any more. In core.clj, do this:
(:import
            [org.apache.hadoop.hbase.filter
            RowFilter
            CompareOp
            RegexStringComparator])

(defn daily-row-filter
  []
  (RowFilter. CompareOp/EQUAL
              (RegexStringComparator. "^[0-9a-fA-F-]{36}:[0-9]{8}$")))

  (hb/with-scanner [results (hb/scan ht
                                     :filter (daily-row-filter) ] ; ...
  )
Hope I can help someone, or at least I'm helping myself of the future.

2013-08-09

MSP430 Lauchpad + Forth 初開始

Prolog: There are tons of articles about Forth on MSP430 in English. I would thus like to write some introductory in Chinese.

小弟長久以來在FigTaiwan論壇上潛水,一直沒有產出,深感抱歉。今晚來寫點介紹性的文章,給軟體業的朋友們參考。

Forth (符式) 是一個 postfix (正式名稱叫逆波蘭表示法) 的雙堆疊語言,詳細的介紹網路上很多了,本篇的重點在如何在 Linux 下,把一個 Forth 環境裝到 MSP430 Lauchpad 上面。

1. 按照這張圖,把 TX/RD 設成硬體 UART:

用 USB 線把 MSP430 Lauchpad 和電腦接起來,dmesg 可以看到以下字樣:

[55571.993498] usb 3-1: Product: Texas Instruments MSP-FET430UIF
[55571.993503] usb 3-1: Manufacturer: Texas Instruments
...
[55571.999657] cdc_acm 3-1:1.0: ttyACM0: USB ACM device
2. 跟著這個網頁的說明,在 Linux 下燒錄第一個 MSP430 的程式 (這邊使用 Ubuntu, 11.x 版之後應該都可以):
$ git clone http://github.com/wendlers/msp430-harduart.git
$ sudo apt-get install mspdebug gcc-msp430
$ make
3. 編譯完成後,就可以燒錄了。
$ sudo make flash-target

mspdebug rf2500 "prog ./bin/firmware.elf"
MSPDebug version 0.19 - debugging tool for MSP430 MCUs
Copyright (C) 2009-2012 Daniel Beer 
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Trying to open interface 1 on 005
Initializing FET...
FET protocol version is 30394216
Configured for Spy-Bi-Wire
Set Vcc: 3000 mV
Device ID: 0x2553
Device: MSP430G2553
Code memory starts at 0xc000
Number of breakpoints: 2
Erasing...
Programming...
Writing  364 bytes to c000 [section: .text]...
Writing  106 bytes to c16c [section: .rodata]...
Writing   32 bytes to ffe0 [section: .vectors]...
Done, 502 bytes written
4. 到這邊一切順利的話,可以先玩玩看測試用的小程式。
$ sudo screen /dev/ttyACM0 9600
如果你和我一樣,看到恆亮的紅綠燈,screen 沒有顯示任何字,可以改用 minicom:
$ sudo apt-get install minicom
$ minicom -b 9600 -D /dev/ttyACM0
然後按下 reset 鍵,就可以看到 Logo 了:
***************
MSP430 harduart
***************

PRESS any key to start echo example ... 
5. 按任意鍵之後,就可以開始打字。每打一個字母,紅燈就會 toggle 一次。

6. 玩得開心之後,按 Ctrl-A x 離開 Minicom 。如果離不開,直接把 USB 線拔掉即可。 如果你想用 gcc-msp430 來寫程式,那到這個步驟就可以了。下面要開始進入 Forth

7. 去 4E4th 下載 Forth 主程式。 4E4th 主程式大概 6K, 也就是說還有 8K 的空間,可以讓你寫自己的程式。

8. 燒錄:

$ sudo mspdebug rf2500 "prog ./4e4th.a43"
MSPDebug version 0.19 - debugging tool for MSP430 MCUs
Copyright (C) 2009-2012 Daniel Beer 
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Trying to open interface 1 on 007
Initializing FET...
FET protocol version is 30394216
Configured for Spy-Bi-Wire
Set Vcc: 3000 mV
Device ID: 0x2553
Device: MSP430G2553
Code memory starts at 0xc000
Number of breakpoints: 2
Erasing...
Programming...
Writing   34 bytes to 1080...
Writing 4096 bytes to e000...
Writing 3482 bytes to f000...
Writing   32 bytes to ffe0...
Done, 7644 bytes written
9. 4E4th 的 Wiki 上有很詳細的說明,不過他們的 PDF 檔不是標準格式,我用 Chrome 打不開,另存新檔後用 evince 就可以看。

10. 奇怪的是,Forth 的範例我沒辦法使用 minicom, 但是用 screen 跑卻沒問題。可以試試這個:

$ sudo screen /dev/ttyACM0

打指令:
red cclr        -- 紅燈滅
red cset        -- 紅燈亮
green cclr      -- 綠燈滅
green cset      -- 綠燈亮

已經可以使用 Forth 指令了:
11 22 33 ok 
. 33 ok 
. 22 ok 
. 11 ok 

奇怪的是,它的堆疊沒有檢查下限?
. 12771 ok
. -3823 ok
. -7228 ok
11. 定義自己的 word 也沒問題:
: TEST ." Hello Forth!" ; ok 
TEST Hello Forth!ok 
12. 為了 MSP430 特製指令在這裡,需要參考一下: Glossary 完整的符式字的定義在這裡。

13. 再多玩一下...

red . . [Enter] 33 1 ok
即 adr = 0x21 (Port1 輸出), m = bit0
green . . [Enter] 33 64 ok 
即 adr = 0x21 (Port1 輸出), m = bit6
14. 這塊板子可以幹什麼呢? 這邊 有些介紹。

15. 如果想研究原始碼,就 clone 一份下來吧!

$ svn co http://www.forth-ev.de/repos/4e4th/

2013-03-08

Access HBase in a map-reduce job (Clojure)

Sometimes a Hadoop cluster is secured and a map-reduce job needs to obtain an authentication token to access HBase. Here's how we do it in Java:
if (User.isSecurityEnabled()) {
    try {
        User.getCurrent().obtainAuthTokenForJob(job.getConfiguration(), job);
    } catch (IOException ioe) {
        LOG.error(job.getJobName()+ ": Failed to obtain current user.");
    } catch (InterruptedException ie) {
        LOG.info(job.getJobName()+ ": Interrupted obtaining user authentication token");
        Thread.interrupted();
    }
}
So, how to do it in Clojure? First, use clojure-hadoop to create a map-reduce job. You may not use (defjob/defjob), because we need to add something before job submission. Second, we have to make Job job = new Job(HBaseConfiguration.create()). An example follows.
(defn- hbase-set-kerberos [#^Job job]
  (if (User/isSecurityEnabled)
    (try
      (.obtainAuthTokenForJob (User/getCurrent) (.getConfiguration job) job)
      (catch IOException e
        (throw (IOException. "Failed to obtain current user.")))
      (catch InterruptedException e
        (throw (InterruptedException. "Interrupted obtaining user authentication token"))))))
 
(defn tool-run [^Tool this args]
  (doto (Job. (HBaseConfiguration/create))
    (.setJarByClass (.getClass this))
    (.setJobName (str "hbfeeder.mrjob: " (second args)))
    (.setMapperClass (Class/forName "hbfeeder.mrjob_mapper"))
    (.setReducerClass (Class/forName "hbfeeder.mrjob_reducer"))
    (.setNumReduceTasks 0)
    (.setInputFormatClass TextInputFormat)
    (.setOutputFormatClass NullOutputFormat)
    (FileInputFormat/setInputPaths ^String (second args))
    (hbase-set-kerberos)       ; THIS LINE PLAYS THE MAGIC
    (.waitForCompletion true))
  0)
A runnable example can be found at clj-hbase-mapper-example on Github.

2012-12-12

Dokuwiki + Slideshow + Syntaxhighlighter 3

Wonder why I can't Google anything like this. I was using Dokuwiki to make netty slides in Linux (as Libreoffice Impress is slow and easy to crash). Everything was fine, just that deck.js doesn't support Syntaxhighlighter 3 to the extent of Dokuwiki. And there is a final strike: DIV size varies according to the content, the fewer text in it, the larger font-size is. Ugly. So I downloaded S5 Reloaded plugin for Dokuwiki and tweaked quite a bit to make it work with Syntaxhighlighter. First, choose a theme for "presentation" only (hence the theme in the wiki is different from that on slides). I chose shThemeDark.css. Vim your syntaxhighlighter3/sxh3/styles/shThemeRDark.css, patch one line:
.syntaxhighlighter {
  background-color: #1b2426 !important;
  font-size: 14px !important;  /* ADD THIS */
}
Or it inherits font-size: 19px from S5. Then, patch s5reloaded/renderer.php.
<link rel="stylesheet" type="text/css" href="/dokuwiki/lib/plugins/syntaxhighlighter3/sxh3/styles/shCore.css"/>
<link rel="stylesheet" type="text/css" href="/dokuwiki/lib/plugins/syntaxhighlighter3/sxh3/styles/shThemeRDark.css"/>
<script type="text/javascript" charset="utf-8" src="/dokuwiki/lib/plugins/syntaxhighlighter3/sxh3/scripts/shCore.js"></script>
<script type="text/javascript" charset="utf-8" src="/dokuwiki/lib/plugins/syntaxhighlighter3/sxh3/scripts/shAutoloader.js"></script>
And put SyntaxHighlighter.autoloader in document_end(). That's all. Enjoy!

2012-11-24

ffmpeg/avconv Parameters for Kindle Fire

Just taking a note for myself:
# For libx264
apt-get install ffmpeg libavcodec-extra-53
# For input < 1024x600
avconv -i INPUT -acodec aac -ab 96k -vcodec libx264 -f mp4 -crf 22 -strict experimental OUTPUT.mp4
# For input > 1024x600
ffmpeg -i INPUT -acodec aac -ab 96k -vcodec libx264 -f mp4 -crf 22 -s 1024x600 -strict experimental OUTPUT
Copied from ConversionMatrix, only that -vpre slow is no longer supported.

2012-08-06

clojure-hbase with filter

The project I'm working on got switched from MySQL to HBase in order to react against some massive prediction of data. Thank David Santiago for his brilliant clojure-hbase project, I can access HBase in Clojure. Here are some tips if you have to work in some unfriendly enterprise environment like I am. After all, Clojure is our rejoice! :)
  • Tip #1. Customised HBase CLASSPATH
Forget about resources/* in your Leiningen project. Just lein jar and make a run.sh:
#!/bin/bash
CP=$(lein classpath)
java -cp "/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/etc/hbase/conf:/usr/lib/hbase/hbase.jar:/usr/lib/hbase/lib/*:./target/*:$CP" myproject.core $*
  • Tip #2. Filter
You may use filter in hb/get and hb/scan. Just follow this, and you get all rows "U-*" whose columns begin with "M-d5".
(defn prefix-filter
  [prefix]
  (ColumnPrefixFilter. (Bytes/toBytes prefix)))

(defn test-scanner
  [^HTable mr]
  (hb/with-scanner [results (hb/scan mr
                                     :start-row "U-"
                                     :stop-row "U."
                                     :filter (prefix-filter "M-d5")
                                     )]
                   (doall (map #(println %)
                               (seq results)))))

2012-04-26

新手的 Neo4j: node_auto_index

跟著 Cypher Cookbook玩耍的時候,遇上了一個怪問題:
ljm:~/neo4j$ ./neo4j-community-1.7/bin/neo4j-shell 
NOTE: Remote Neo4j graph database service 'shell' at port 1337
Welcome to the Neo4j Shell! Enter 'help' for a list of commands

neo4j-sh (0)$ START a=node:node_auto_index(name="Thomas Anderson") RETURN a
MissingIndexException: Index `node_auto_indedex` does not exist
啥? 我 Google 了一下,發現要把 neo4j.properties 裡面的 node_auto_indexing 打開,所以:
node_auto_indexing=true
node_keys_indexable=name,age
relationship_auto_indexing=true
relationship_keys_indexable=IS_ROOT_OF,KNOWS,CODED_BY
咦? 還是沒有用? 再 Google 了一下,發現打開後 **新增的 nodes** 才有 index,因為它是 lazy index。 把 nodes 砍掉重練,於是可以用 Cypher 了... 試試看,很好玩!
neo4j-sh (0)$ START a=node:node_auto_index(name="Cypher"),
b=node:node_auto_index(name="Thomas Anderson")
MATCH p=shortestPath(a-[r*..10]-b)
RETURN p     
+--------------------------------------------+
| p                                          |
+--------------------------------------------+
| (18)<--[KNOWS,22]--(16)<--[KNOWS,19]--(15) |
+--------------------------------------------+
1 row, 1 ms
要從 neo4jrestclient 來 query 也可以:
>>> c = gdb.extensions.CypherPlugin
>>> r = c.execute_query("START a=node:node_auto_index(name='Trinity') MATCH a-->b RETURN a,b")
>>> [d['self'] for d in r['data'][0]]
[u'http://localhost:7474/db/data/node/30', u'http://localhost:7474/db/data/node/32']
>>> [d['data'] for d in r['data'][1]]
[{u'name': u'Trinity'}, {u'age': 29, u'name': u'Thomas Anderson'}]
>>> [d['data'] for d in r['data'][0]]
[{u'name': u'Trinity'}, {u'lastname': u'Reagan', u'name': u'Cypher'}]
再列一些參考資料:

Histogram in Clojure

學到 (rand-int) 的時候,想說應該驗證一下亂數產生器的分佈夠不夠均勻,可是不知道怎麼寫 histogram. 下面的程式參考了 Along Came Betty:
(def randmap (for [x (range 100)] (rand-int 10)))
==> #'user/randmap
(reduce (fn [m w] (assoc m w (+ 1 (m w 0)))) {} randmap)
==> {0 9, 1 7, 2 11, 3 14, 4 9, 5 7, 6 11, 7 11, 8 9, 9 12}
覺得 0 1 2 3 ... 礙眼?
(vals (reduce (fn [m w] (assoc m w (+ 1 (m w 0)))) {} randmap))
==> (9 7 11 14 9 7 11 11 9 12)
嗯,看來這個亂數產生器不夠靠譜 XD

2012-04-25

新手的 Neo4j: plugins

去 neo4j-server-examples 下載常用的 plugin ,然後重新啟動 neo4j-service。 好了,可以接上 這一頁 的範例了...
>>> from neo4jrestclient.client import GraphDatabase
>>> gdb = GraphDatabase("http://localhost:7474/db/data/")
>>> gdb.extensions
{u'CypherPlugin': <Neo4j ExtensionModule: [u'execute_query']>, u'GremlinPlugin': <Neo4j ExtensionModule: [u'execute_script']>, u'GetAll': <Neo4j ExtensionModule: [u'get_all_nodes', u'getAllRelationships']>}


>>> gdb.node[5].properties
{u'name': u'Trinity'}
>>> gdb.node[9].properties
{u'name': u'The Architect'}
>>> trinity = gdb.node[5]
>>> # 找出從 Trinity 到 The Architect 的最短路徑
>>> p = trinity.extensions.ShortestPath.shortestPath(gdb.node[9])
>>> p
[{u'relationships': [u'http://localhost:7474/db/data/relationship/8', u'http://localhost:7474/db/data/relationship/9', u'http://localhost:7474/db/data/relationship/10'], u'start': u'http://localhost:7474/db/data/node/5', u'nodes': [u'http://localhost:7474/db/data/node/5', u'http://localhost:7474/db/data/node/7', u'http://localhost:7474/db/data/node/8', u'http://localhost:7474/db/data/node/9'], u'end': u'http://localhost:7474/db/data/node/9', u'length': 3}]
>>> print [gdb.node[n].properties['name'] for n in p[0]['nodes']]
[u'Trinity', u'Cypher', u'Agent Smith', u'The Architect']

2012-04-24

新手的 Neo4j + Python

因為我在公司使用 Ubuntu 11.10, 參考這一頁安裝時,需要留意一下... 不需要安裝 Neo4j community 1.7。 如果要安裝 neo4j-embedded 的話:
  1. 直接 sudo apt-get install python-jpype ,不用下載了。
  2. 如果一定要下載來自己編譯,可以用 sudo JAVA_HOME=/usr/lib/jvm/java-6-sun python setup.py install
  3. sudo apt-get install python-pip
  4. sudo pip install neo4j-embedded
就可以開始玩了... 網路上有好多 demo, 寫得好複雜的樣子,晚一點再來看有沒有簡單一點的。 啊。搞錯了。 用 neo4j-embedded 雖然沒什麼問題,但是公司的應用必須有 standalone server,而且把服務跑起來,才能用 browser,所以要安裝 neo4j-community:
  1. sudo ./neo4j-community-1.7/bin/neo4j install 安裝服務
  2. sudo -u neo4j /etc/init.d/neo4j-service start 把Neo4j打開
  3. 用瀏覽器看 http://localhost:7474/webadmin/
Python 可以用的 RESTful client 有兩個: neo4jrestclient 和 py2neo。 公司的 VM 上要安裝新軟體很痛苦,所以我本來應該裝 py2neo,可是因為 neo4jrestclient 包裝得和 neo4j-embedded 很像,所以下面的例子用的是 neo4jrestclient:
# 有名的 The Matrix 的例子
from neo4jrestclient.client import GraphDatabase
db = GraphDatabase("http://localhost:7474/db/data/")

thomas = db.node(name = 'Thomas Anderson', age = 29)
trinity = db.node(name = 'Trinity')
morpheus = db.node(name = 'Morpheus', rank = 'Captain', occupation = 'total badass')
cypher = db.node(name = 'Cypher', lastname = 'Reagan')
agentSmith = db.node(name = 'Agent Smith', version = '1.0b', language = 'C++')
architect = db.node(name = 'The Architect')
thomas.KNOWS(trinity)
trinity.LOVES(thomas, since = 'meeting the oracle')
thomas.KNOWS(morpheus)
trinity.KNOWS(cypher, since = 'a year before the movie', cooperatesOn = 'The Nebuchadnezzar')
cypher.KNOWS(agentSmith, disclosure = 'secret')
agentSmith.CODEDBY(architect)
搞定後,就可以用 python 命令列來玩耍:
>>> from neo4jrestclient.client import GraphDatabase
>>> gdb = GraphDatabase("http://localhost:7474/db/data/")
>>> n = gdb.node[5] # 假設 Trinity 是 node[5] 可以去 browser 看 :)
>>> n.relationships.all()[:]
[<Neo4j Relationship: http://localhost:7474/db/data/relationship/8>, <Neo4j Relationship: http://localhost:7474/db/data/relationship/5>, <Neo4j Relationship: http://localhost:7474/db/data/relationship/6>]
>>> n.relationships.all()[0].type
'KNOWS'
>>> n.relationships.all()[0].start
<Neo4j Node: http://localhost:7474/db/data/node/5>
>>> n.relationships.all()[0].end
<Neo4j Node: http://localhost:7474/db/data/node/7>
>>> n.relationships.all()[0].properties
{u'since': u'a year before the movie', u'cooperatesOn': u'The Nebuchadnezzar'}
然後是 traversals:
>>> from neo4jrestclient import client
>>> n.traverse(types=[client.All.KNOWS])[:]
[<Neo4j Node: http://localhost:7474/db/data/node/7>, <Neo4j Node: http://localhost:7474/db/data/node/4>]
>>> n.traverse(types=[client.Outgoing.KNOWS])[:]
[<Neo4j Node: http://localhost:7474/db/data/node/7>]
>>> n.traverse(types=[client.Incoming.KNOWS])[:]
[<Neo4j Node: http://localhost:7474/db/data/node/4>]
比方說想找 Trinity KNOWS 的人 KNOWS 的... (我為了舉例多加了一些關係)
# Ruby 寫起來好漂亮 T_T ==> me.outoging(:frienship).outgoing(:datingship)
>>> [n.traverse(types=[Outgoing.KNOWS])[:] for n in trinity.traverse(types=[Outgoing.KNOWS])[:]]
[[<Neo4j Node: http://127.0.0.1:7474/db/data/node/8>], [<Neo4j Node: http://127.0.0.1:7474/db/data/node/5>, <Neo4j Node: http://127.0.0.1:7474/db/data/node/4>]]
# 也可以接上 neo4j 的範例...
>>> gdb.traversal().relationships('KNOWS').traverse(trinity).nodes[:]
[<Neo4j Node: http://127.0.0.1:7474/db/data/node/7>, <Neo4j Node: http://127.0.0.1:7474/db/data/node/6>, <Neo4j Node: http://127.0.0.1:7474/db/data/node/4>]
# Incoming 和 Outgoing 直接偷偷用關鍵字...
>>> gdb.traversal().relationships('KNOWS', 'out').traverse(trinity).nodes[:]
[<Neo4j Node: http://127.0.0.1:7474/db/data/node/7>, <Neo4j Node: http://127.0.0.1:7474/db/data/node/6>]
>>> gdb.traversal().relationships('KNOWS', 'in').traverse(trinity).nodes[:]
[<Neo4j Node: http://127.0.0.1:7474/db/data/node/4>]

2012-04-19

Migrate "Simple Web App" to Clojure 1.3.0

Another trial to migrate Mark McGranaghan's awesome example of Developing and Deploying a Simple Clojure Web Application to Clojure 1.3.0. First, project.clj: remove clojure-contrib and refresh the revisions...
defproject adder "0.0.1"
  :description "Add two numbers."
  :dependencies
  [[org.clojure/clojure "1.3.0"]
   [ring/ring-core "1.1.0-RC1"]
   [ring/ring-devel "1.1.0-RC1"]
   [ring/ring-jetty-adapter "1.1.0-RC1"]
   [compojure "1.0.2"]
   [hiccup "1.0.0-RC2"]]
  :dev-dependencies
  [[lein-run "1.0.0-SNAPSHOT"]])
hiccup.page-helper became hiccup.page, and we have to add ring.middleware.params after Compojure 0.6.0!!! (Really painful to find out why POST isn't working anymore.) In core.clj:
(:use [compojure.core]
      [hiccup.core]
      [hiccup.page]
      [ring.middleware.reload]
      [ring.middleware.stacktrace]
      [ring.util.response]
      [ring.middleware.file]
      [ring.middleware.file-info]
      [ring.middleware.params :only [wrap-params]])

; and for app ...
(def app
  (-> #'handler
      (wrap-params handler)
      (wrap-utf)
      (wrap-file "public")
      (wrap-file-info)
      (wrap-request-logging)
      (wrap-if development? wrap-reload '[adder.middleware adder.core])
      (wrap-bounce-favicon)
      (wrap-if production?  wrap-failsafe)
      (wrap-if development? wrap-stacktrace)))
Others are just like they were in Mark's brilliant tutorial. The source code is available here.

Migrate "REST" to Clojure 1.3.0 with Slingshot

I'm really inspired by Mark McGranaghan's Building REST APIs for Clojure Web Applications. It uses Clojure 1.2.0, but I'm really fancy at new things and wanted to port the code to Clojure 1.3.0. I learned that clojure.contrib is no longer in use and clojure.contrib.condition is now replaced by Steve Gilardi's Slingshot. So I hacked a bit ... I'm not sure if my codes are good enough. Comments are more than welcome. Changes in web.clj:
  (:use [compojure.core]
        [ring.middleware.json-params]
        [slingshot.slingshot :only [try+]])
; ...
(defn wrap-error-handling [handler]
  (fn [req]
      (try+
       (or (handler req)
           (json-response {"error" "resource not found"} 404))
       (catch JsonParseException e
              (json-response {"error" "malformed json"} 400))
       (catch [:type :bad-req] {:keys [reason message]}
              (json-response {"error" message} (error-codes reason))))))
Changes in elem.clj
  (:use [slingshot.slingshot :only [throw+]])
; ...
(defn get [id]
  (or (@elems id)
      (throw+ {:type :bad-req :reason :not-found
              :message (format "elem '%s' not found" id)})))

(defn put [id attrs]
  (if (empty? attrs)
      (throw+ {:type :bad-req :reason :invalid
              :message "attrs are empty"})
; ...
And it passes test.sh: (Sorry! I don't know how to write lein test cases.)
{}
{"error":"elem '1' not found"}
{"group":"bar"}
{"group":"biz","tag":"bat"}
{"group":"bar"}
{"2":{"group":"biz","tag":"bat"}}
{"error":"elem '3' not found"}
{"error":"resource not found"}
{"error":"malformed json"}400
{"error":"elem '3' not found"}404
{"error":"attrs are empty"}400
You may download the example here.

2012-04-14

Clojure

大部份資料都 Google 得到,這裡只是做個筆記。 下載 1.3.0 版後,直接解開來就可以用了。當然如果安裝 lein 會更優。lein 會把所有的東西都放在自己的目錄裡,所以在這家公司也沒問題。
C:\download\clojure-1.3.0>java -jar clojure-1.3.0.jar
Clojure 1.3.0
user=> (println "Hello World")
Hello World
nil
user=> (javax.swing.JOptionPane/showMessageDialog nil "Hello World")
nil
user=>
資料: 筆記:
; System.out.println()
(println "Hello World")
; Swing
(javax.swing.JOptionPane/showMessageDialog nil "Hello World")
;
; Name space
;
(def v1 10)     ; user/v1
(in-ns 'myapp)
(def v1 100)    ; myapp/v1
(in-ns 'user)
v1              ; = 10
myapp/v1        ; = 100
;
; Polymorphism
;
(defn force-it
    "The first function a young jedi needs"
    ([]
        (str "Use the force, Luke"))
    ([jedi]
        (str "Use the force, " jedi)))
;
; Lambda
;
(map (fn [x] (* 2 x)) v)  ; v = (2 4 6 8)
(map #(* 2 %) v)          ; #() = lambda
;
;
;
;
;
;
(take 10 (cycle [1 2 3 4]))
; ==> (1 2 3 4 1 2 3 4 1 2)
(interleave [:a :b :c :d :e] [1 2 3 4 5])
; ==> (:a 1 :b 2 :c 3 :d 4 :e 5)
(partition 3 [1 2 3 4 5 6 7 8 9 10])
; ==> ((1 2 3) (4 5 6) (7 8 9))
(map vector [:a :b :c :d :e] [1 2 3 4 5])
; ==> ([:a 1] [:b 2] [:c 3] [:d 4] [:e 5])
(apply str (interpose \, "asdf"))
; ==> "a,s,d,f"
(interpose \, "asdf")
; ==> (\a \, \s \, \d \, \f)
(System/getProperties)
; ==> #<Properties {java.runtime.name=Diablo Java(TM) SE Runtime Environment, sun.boot.library.path=/usr/local/diablo-jdk1.6.0/jre/lib/amd64, ...
(first (System/getProperties))
; ==> #<Entry java.runtime.name=Diablo Java(TM) SE Runtime Environment>
(.getBytes "Hello")
; ==> #<byte[] [B@5eea3cdf>
(first (.getBytes "Hello"))
; ==> 72
(re-seq #"\w+" "the quick brown fox")
; ==> ("the" "quick" "brown" "fox")
(sort (re-seq #"\w+" "the quick brown fox"))
; ==> ("brown" "fox" "quick" "the")
(java.io.File. ".")
; ==> #<File .>
(take 5 (file-seq (java.io.File. ".")))
; ==> (#<File .> #<File ./.cshrc> #<File ./.login> #<File ./.mail_aliases> #<File ./.login_conf>)
(count (file-seq (java.io.File. ".")))
; ==> 16760
(with-open [rdr (reader "README")] (count (line-seq rdr)))
; ==> CompilerException java.lang.RuntimeException: Unable to resolve symbol: reader in this context, compiling:(NO_SOURCE_PATH:5)
(use '[clojure.contrib.repl-utils :only (show)])
; ==< nil
(show java.util.Collections "sort")
; ==< ===  public java.util.Collections  ===
; ==< [11] static checkedSortedMap : SortedMap (SortedMap,Class,Class)
; ==< [12] static checkedSortedSet : SortedSet (SortedSet,Class)
; ==< [40] static sort : void (List)
; ==< [41] static sort : void (List,Comparator)
; ==< [47] static synchronizedSortedMap : SortedMap (SortedMap)
; ==< [48] static synchronizedSortedSet : SortedSet (SortedSet)
; ==< [53] static unmodifiableSortedMap : SortedMap (SortedMap)
; ==< [54] static unmodifiableSortedSet : SortedSet (SortedSet)
(slurp "readme.txt")
; ==< 讀出 readme.txt 的內容
(use '[clojure.contrib.duck-streams :only (read-lines)])
; ==< nil
(doseq [line (read-lines "readme.txt")] (println line))
; ==< 印出 readme.txt 的內容
(java.io.File. "readme.txt")
; ==< #>File readme.txt<
(reader (java.io.File. "readme.txt"))
; ==< #>BufferedReader java.io.BufferedReader@42db681c<
(line-seq (reader (java.io.File. "readme.txt")))
; ==< 印出 readme.txt 的內容
(with-open [rdr (java.io.BufferedReader. (java.io.FileReader. "readme.txt"))] (doseq [line (line-seq rdr)] (println line)))
; ==< 印出 readme.txt 的內容
(doc line-seq)
; ==< java.io.BufferedReader 的 lazy-seq!
(def book {:title "1Q84" :author "村上春樹"})
; ==< #'user/book
(assoc book :publisher "新潮社")
; ==< {:publisher "新潮社", :title "1Q84", :author "村上春樹"}
book
; ==< {:title "1Q84", :author "村上春樹"}

Jython, Jyson / simplejson, SQLite

因為公司的開發用 VM 限制重重,前輩說他們覺得最優的方式,就是在自己的目錄下放一堆 JAR ... well, 還好 Python 也做得到。
  • Jython - 直接安裝在自己的目錄下,沒問題。
  • JSON - 使用 Jyson. 他們的首頁壞掉了,可以在 這裡下載 ,或是使用 simplejson
  • SQLite3 - 用 JDBC 沒問題,我用 Xerial 版而不是 Zentus 版。 這裡有範例, 測試過了可以用。
  • HBase - 還沒試,晚一點更新上來
有這堆限制、卻又很不想學 Java ,或許該來看看 Clojure ...

2012-04-11

Ubuntu 11.10 的遠端桌面; JSON for Python 2.4

官網上寫了很多方法,我試過 vino, XDMCP, x11vnc, 最後以 x11vnc 勝出。 直接安裝就好了,按這裡的寫法,如果你已經登錄了,就用 ssh 開一個視窗:
x11vnc -display :0
如果你還沒登錄,一樣用 ssh 開一個視窗:
sudo x11vnc -auth /var/run/lightdm/root/:0 - display :0
建議使用 x11vnc -storepasswd 設定密碼,不然就要記得擋掉 port 5900 的外部存取,然後開 SSH Tunnel 。 客戶端我試用了 remmina, 沒有想像的方便、好用。 新的公司有些文化因素,使得開發人員沒辦法使用比較新的程式語言,安裝模組也會遇到很大的阻力。隨手記一下在 Python 2.4 / 2.6 都能用 JSON 模組的方法: (從 這裡 看到的)
try:
    import json
except ImportError:
    import simplejson as json
補充: 如果沒辦法編譯 simplejson 的 C 加速模組也沒關係,改一下 setup.py 不要讓它 build, 還是可以用單純用 Python 寫的模組。 後記:Python 2.6 的 json 模組實在太慢了... 預設使用 simplejson 比較快。

2012-04-05

SBCL for CentOS 6.2 x64

單純是想在 CentOS 6.2 x86_64 上安裝 SBCL 時遇上了一些問題。 SBCL 下載頁 裡的 AMD64 不能直接用,它需要 /lib64 裡的 GLIBC 2.14,但 CentOS 6.2 附的是 2.12。 解法如下:
  1. 下載 x86 版的 binary ,安裝。這版也可以直接用。
  2. 下載原始碼,編譯。
自己編譯的 AMD64 版在 make test 時會有一個 error (和多緒有關的),不知道會有什麼影響...

2012-03-28

給懂 C++ 的人的 Java 入門

最近因為轉換跑道,需要學 Java ,可是市面上的書大多是從頭教起的。 Java for C++ Programmers 是給已經懂 C++ 和基本的資料結構的人用的線上教學。值得一看!