[PyConJP2012] Problems and solutions about internal DSL design in Python. PythonでDSLを設計する上での問題点と解決策。 Video: http://www.youtube.com/watch?v=l8ptNmtB0G8

1. PyConJP 2012
Fantastic DSL
in Python
Makoto Kuwata <kwa@kuwata-lab.com>
http://www.kuwata-lab.com/
2012-09-16 (Sun)
Video: http://www.youtube.com/watch?v=l8ptNmtB0G8

2. Agenda
•About DSL
•Fantastic 'with' statement
•Fantastic 'for' statement
•Fantastic Decorator
•Fantastic Operator

3. About DSL

4. What is DSL (Domain Specific Language?)
DSL
Google Search

5. Internal DSL v.s. External DSL
•Internal DSL (内部DSL)
•Written inside an existing host language
既存の言語を使って、その言語の文法を使って書かれたDSL
•No need to implement parser
パーサを書く必要なし
•Some lang are not good at internal DSL
言語によっては苦手な場合がある
•External DSL (外部DSL) Python?
•Original, indendent syntax
独自の文法
•Need to implement parser
パーサを実装する必要あり
•All languages have same power about external DSL
言語による得手不得手は、ほぼない

6. Fantastic 'with' Statement

7. What is 'with' statement?
•Enforces finishing ## Python 2.4
process f = open('file')
終了処理を強制する機能 try:
•ex: close opened file text = f.read()
certainly finally:
例：ファイルを必ず閉じる f.close()
## Python 2.5 or later
with open('file') as f:
text = f.read()

8. How to use 'with' statement in DSL?
•Appends pre-
Append pre-process
process and post- 前処理を追加
process around
}
block x=1
ブロックに前処理と後処理をくっ
y=2 code block
つける z=3
Append post-process
後処理を追加

9. Case Study: Change Directory (Kook)
## before ## after
chdir('build') with chdir('build'):
cp('../file', 'file') cp('../file', 'file')
system('cmd...') system('cmd...')
chdir('..')

10. Case Study: Start/Stop time (Benchmarker)
## before ## after
t1 = time.time() bm = Benchmarker()
for i in xrange(N): with bm('func'):
func() for i in xrange(N):
t2 = time.time() func()
print('func: %s'
% (t2-t1))

11. Case Study: Form Builder
## before
<form action="/" method="POST">
<input type="hidden" name="_csrf"
value="${request.csrf()}" />
<input type="text" ... />
<input type="submit" ... />
</form>

12. Case Study: Form Builder
## after
% with FormBuilder(request, "/") as fb:
<input type="text" ... />
<input type="submit" ... />
% endwith

13. Pitfall: Variable Scope
with test("arithmetic operation"):
with test("1+1 should be 1"):
x = 1+1
assert x == 2
with test("1-1 should be 0"):
x = 1-1 Identical variable because
assert x == 0 scope is not independented
スコープが独立してないので変数も共有される

14. Pitfall: Cannot Skip Block!
<div>
% with cache("key", 60*60*1):
<ul>
% for item in get_items():
<li>${item}</li>
% endfor You may want to skip block
</ul> when cache is not expired,
% endwith but Python doesn't allow it!
</div> キャッシュが生きている間はブロックを
スキップしたいんだけど、実はできない

15. Fantastic 'for' Statement

16. What is 'for' statement?
•Iterates code block ## repeat 100 times
ブロックを繰り返す機能 for i in xrange(100):
•ex: repeat block for N print("i=%s" % i)
times
例：ブロックをN回実行 ## read lines from file
with open('file') as f:
•ex: read each line for line in f:
from opened file
print(line)
例：オープンしたファイルから
1行ずつ読み込む

17. How to use 'for' statement in DSL? #1
•Appends pre-
Append pre-process
process and post- 前処理を追加
process around
}
iteration block x=1
繰り返し用のブロックに
y=2 iteration block
前処理と後処理をくっつける z=3
Append post-process
後処理を追加

18. How to use 'for' statement in DSL? #1
•Appends pre-
def gfunc():
process and post-
process around
iteration block
....
.... }pre-process
繰り返し用のブロックに for i in xrange(n):
} exec block
前処理と後処理をくっつける
yield
....
.... } post-process

19. Case Study: Start/Stop time (Benchmarker)
## before ## after
t1 = time.time() bm = Benchmarker(
for i in xrange(N): repeat=N)
func1() for _ in bm('func'):
t2 = time.time() func1()
print('func: %s'
% (t2-t1))

20. How to use 'for' statement in DSL? #2
•Iterate code block
def gfunc():
only once
ブロックを1回だけ繰り返す
....
.... } pre-process
yield } exec block
(only once!)
•Emulates 'with' stmt ....
by 'for' stmt .... } post-process
with文をfor文でエミュレート

21. Case Study: Change Directory (Kook)
## before ## after
chdir('tmp') for _ in chdir('tmp'):
cp('../file', 'file') cp('../file', 'file')
system('cmd...') system('cmd...')
chdir('..')
Use 'for' as alternative
of 'with' in Python 2.4
Python2.4では 'with' が使えない
ので、代わりとして 'for' を使う

22. How to use 'for' statement in DSL? #3
•Iterate block in 0 or
def gfunc():
1 time
ブロックを0回または1回だけ
繰り返す
....
.... }pre-process
if condition:
}
•Emulates 'if' stmt by
'for' stmt to skip yield exec block
block according to ....
condition
if文をfor文でエミュレートし、条
.... } post-process
件によってブロックをスキップ

23. Case Study: Fragment Cache (Tenjin)
<div>
% for _ in cache("key", 60*60*1):
<ul>
% for item in get_items():
<li>${item}</li>
% endfor You can skip block when
</ul> cache is not expired! Wow!
% endfor with文と違い、キャッシュが生きている
</div> ときにブロックをスキップできる! ステキ!

24. Case Study: Java and JSP
<div>
<% for (int _: cache("key",60)) { %>
<ul>
<% for (Item item: getItems()) { %>
<li>${item}</li>
<% } %>
</ul>
<% } %>
</div>

25. Fantastic Decorator

26. What is Decorator?
•Higher-order function class Hello(object):
to manipulate
declaring function or @classmethod
class def fn(cls):
宣言中の関数やクラスを操作する print("Hello")
高階関数
•ex: @classmethod(), ## above is same as
@property() def fn(cls):
例：@classmethod(),
print("Hello")
@property fn = classmethod(fn)

27. How to use Decorator in DSL? #1
•Mark function with
def deco(func):
or without args
func._mark = True
関数に目印をつける
return func
def deco_with(arg):
def deco(func):
func._mark = arg
return func
deco

28. Case Study: Marking and Options (Kook)
@recipe
def test(c):
system("python -m oktest test")
@recipe("*.o")
@ingreds("$(1).c", if_exist="$(1).h")
def file_o(c):
system(c%'gcc -o $(product) $(ingred)')

29. How to use Decorator in DSL? #2
•Syntax sugar to
def fn(arg):
pass function arg
print(arg)
引数として関数を渡すときの
シンタックスシュガー func(x, y, fn)
@func(x, y)
def fn(arg):
print(arg)

30. Case Study: Event Handler
## before ## after
def _(): @button.onclick
do() def _():
some() do()
thing() some()
button.onclick(_) thing()
More readable because
@onclick appears first
ブロックより先に@onclickが
きているのでより読みやすい

31. Case Study: Database Transaction
## before ## after
def _(): @transaction
do() def _():
some() do()
thing() some()
transaction(_) thing()
More readable because
@transaction appears first
ブロックより先に@transactionが
現れるので、より読みやすい

32. Case Study: unittest
## before ## after
def _(): @assertRaise2(Err)
do() def _():
some() do()
thing() some()
assertRaise(Err, _) thing()
More readable because
assertRailse() appears first
assertRaises2()のほうがブロックより
先に現れるので、より読みやすい

33. How to use Decorator in DSL? #3
•Alternative syntax of
with func(arg) as x:
with- or for-
....
statement
with文やfor文の代替
•Independent scope!
独立したスコープ!
@func(arg)
•Nested scope! def _(x):
入れ子になったスコープ! ....

34. Case Study: open()
## before ## after
with open('file', @open('file', 'w')
'w') as f: def _(f):
f.write("....") f.write("....")

35. How to use Decorator in DSL? #4
•Alternative syntax of
class Foo(object):
class definition
def meth1(self):
class定義の代替
....
•More natural
representation of
inheritance structure
@classdef
継承構造のより自然な表現
def _():
•More natural scope @methoddef
より自然なスコープ
def _():
....

36. Class Inheritance
class Parent(object):
....
class Child(Parent):
....
Inheritance structure
class Baby(Child): is not represented
.... visually
継承構造が視覚的には
表現されていない

37. Nested Class Definition
class Parent(object):
class Child(Parent):
Represents
class Baby(Child): inheritance structure
.... visually, but not
possible in Python!
こう書けば継承構造が視
覚的に表現できるけど、
Pythonではこう書けない!

38. Nested Structure by with-statement
with define('Parent'):
with define('Child'):
with define('Baby'): Represents structure
.... visually, but scopes
are not independent
構造は視覚的に表現でき
ているが、スコープが独
立していない (共有される)

39. Nested Structure with Decorator
@defclass
def Parent():
@defclass
def Child(): Represents structure
visually, and scopes
@defclass are not shared
def Baby(): 継承構造を視覚的に表現
できるし、スコープも共
.... 有されない (独立している)

40. Strange Scope Rule (for beginner) in Class Definition
1: class Outer(object):
2: VAR = 1
3:
4: class Inner(object):
5: # error
6: print(VAR) Why I can't access
7: # error, too to outer variable?
8: print(Outer.VAR) どうして外側の変数に
アクセスできないの？

41. Natural Scope Rule (for beginner) with Function
1: def outer():
2: VAR = 1
3:
4: def middle():
5: print(VAR) # OK More natual
6: scope rule
7: def inner(): (especially for beginner)
8: print(VAR) # OK より自然なスコープルール
(特に初心者にとっては)

42. Case Study: RSpec
describe("Class1") do
describe("#method1()") do
it("should return None") do
Class1.new.method1().should be_nil
end
end
end

43. Case Study: RSpec
class Class1Test(TestCase):
class method1Test(Class1Test):
def test_should_return_None(self):
assert Class1().method1() is None

44. Case Study: RSpec
@describe("Class1")
def _():
@describe("#method1()")
def _():
@it("should return None")
def _(self):
assert Class1().method1() is None

45. Case Study: RSpec
@describe("Class1")
def _(): Variables in outer are...
def this(): 外側の関数で定義された変数に...
return Class1()
@describe("#method1()")
def _():
@it("should return None")
def _(self):
assert this().method1() is None
Accessable from inner!
内側の関数から自然にアクセスできる!

46. Fantastic Operator

47. Operator Override
1: class Node(object):
2: def __init__(self, op, left, right):
3: self.op, self.left, self.right =
4: Overrides '==' operator op, left, right
5:
6: def __eq__(self, other):
7: return Node('==', self, other)
8: def __ne__(self, other):
9: return Node('!=', self, other)
Overrides '!=' operator

48. Expression Evaluation v.s. AST
Expression Evaluation
x+1 2
AST (Abstract Syntax Tree)
+
x+1
x 1
参考: http://www.slideshare.net/kwatch/dsl-presentation

49. Case Study: Modern O/R Mapper (SQLAlchemy)
Python SQL
==
x == 1 x=1
x 1
==
x == None x is null
x None

50. Case Study: Assertion (Oktest)
Assertion When Failed
## Python AssertionError:
assert x == y (no message)
## Nose AssertionError:
ok_(x == y) (no message)
## Oktest AssertionError:
ok (x) == y 1 == 2 : failed
Shows actual & expected values
失敗時に、実際値と期待値を表示してくれる

51. Case Study: Assertion (Oktest)
Oktest Oktest() returns AssertionObject
ok() はAssertionObject のインスタンスを返す
>>> ok (1+1)
<oktest.AssertionObject object>
>>> ok (1+1).__eq__(1)
Traceback (most recent call last):
...
AssertionError: 2 == 1 : failed.
Overrides '==' operator
'==' 演算子をオーバーライドしている
参考: http://goo.gl/C6Eun

52. Case Study: String Interpolation (Kook)
CMD = 'rst2html -i utf-8'
@recipe('*.html')
@ingreds('$(1).txt') Low readability
def file_html(c): 可読性が低い
system("%s %s > %s" %
(CMD, c.ingreds[0], c.product))
# or
cmd,prod,ingred = CC,c.product,c.ingreds[0]
system("%(cmd)s %(ingred)s > %(prod)s" %
locals())
Name only, No expression
名前のみで式が使えない

53. Case Study: String Interpolation (Kook)
http://www.kuwata-lab.com/kook/pykook-users-guide.html#cookbook-prod
CMD = 'rst2html -i utf-8'
@recipe('hello')
@ingreds('hello.c')
def compile(c):
prod, ingreds = c.product, c.ingreds
system(c%"$(CMD) $(ingreds[0]) > $(prod)")
Operator Indexing
override Both global and available
演算子オーバーライド local var available 添字も利用可能
グローバル変数とローカル
変数の両方が利用可能

54. Case Study: Method Chain Finishing Problem
AppointmentBuilder()
.From(1300)
.To(1400)
.For("Dental")
.done()
Method chain requires end of chain.
メソッドチェーンでは、チェーンの終わりを明示する必要がある

55. Case Study: Method Chain Finishing Problem
- AppointmentBuilder()
.From(1300)
.To(1400)
.For("Dental")
Unary operator works at end of chain!
単項演算子はチェーンの終わりに実行される！
Very similar to unordered list or bullet!
見た目が箇条書きそっくり！

56. Reflected operator
class Foo(object):
def __add__(self, other):
... foo + 1
def __radd__(self, other):
... 1 + foo

57. Reflected operator
def __add__(self, other):
...
int.__add__ = __add__
1 + Foo() Not possible ;<
Pythonではできない
class Foo(object):
def __radd__(self, other):
... No problem :)
1 + Foo() 問題なし

58. Case Study: Should DSL
http://www.should-dsl.info/
def test1(self):
self.player |should| have(11).cards
def test2(self):
self.player.name |should|
equal_to('John Doe')
Instead of adding method to any object,
use bit operator which has low priority
ビットOR演算子の優先順位が低いことを利用して、任意のオブジェクトに
メソッドを追加できないというPythonの欠点を克服した、絶妙なDSL

59. One more thing...

60. DSL設計者に贈る言葉
「キモイ」は褒め言葉！
古参のイチャモンは新規性の証し！
オレが書きたいように書けないのはPythonが悪い！

61. おしまい