Python の新しい正規表現ライブラリ

Python の新しい正規表現ライブラリがすごい

• Perl6と.NETにしかできない(俺知っている限り)グループのなかのサブグループのデータが取れる

>>> m = regex.search(r"(\w{3})+", "123456789")

>>> m.captures(1)
['123', '456', '789']
>>> m.starts(1)
[0, 3, 6]
>>> m.ends(1)
[3, 6, 9]
>>> m.spans(1)
[(0, 3), (3, 6), (6, 9)]

• 長さ一定でないLookBehindができる。
• .finditer と同様に .splititer ができる。
• Posix character classが追加された。

[[:alpha:]], [[:^alpha:]]

• マッチしたグループの名前でとれる。たぶんnamed tuple使っていると思う。

>>> m = regex.search(r"(?P.*?)(?P\d+)(?P.*)", "pqr123stu")
>>> print m["before"]
pqr
>>> print m["num"]
123
>>> print m["after"]
stu
>>> print len(m)
4
>>> print m[:]
('pqr123stu', 'pqr', '123', 'stu')

• \N{name} という表現ができる
• Unicode関係の\P{name} という表現ができる。Unicode 6.0まで対応しているみたい
• 逆方向で検索することができる

>>> regex.findall(r"..", "abcde")
['ab', 'cd']
>>> regex.findall(r"(?r)..", "abcde")
['de', 'bc']

• それ以外にもいろんな機能が追加されたそうです....

スピードも１割ぐらいは早くなったと思う。re2に勝てないけどね。

ただこういうKiller Featuresあるかぎりみんな使い続けるでしょう　。。。

詳しくは　http://pypi.python.org/pypi/regex　で

Memo# - Regular Expressions - Part 2 - Substitutions with Grouping

Test String

7800000 (7.8M Characters)

"abbbcdexfbeczexczczkef111anncdehbzzdezf" * 200000

Very Simple Replace

regex pattern: bc , Replace with : TEST , mine is 2.5x faster

my regex sub 0.140000104904 s 8200000 abbTESTdexfbeczexczczkef111anncdehbzzdezfabbTESTdexfbeczexczczkef111anncde....

python's re sub 0.359999895096 s 8200000 abbTESTdexfbeczexczczkef111anncdehbzzdezfabbTESTdexfbeczexczczkef111anncde....

With Range Matching

regex pattern: [a-z]c , Replace with : TEST , mine is 6x faster

my regex sub 0.453000068665 s 9800000 abbTESTdexfbTESTzeTESTTESTzkef111anTESTdehbzzdezfabbTESTdexfbTESTze....

python's re sub 2.82800006866 s 9800000 abbTESTdexfbTESTzeTESTTESTzkef111anTESTdehbzzdezfabbTESTdexfbTESTze....

With Range Matching + Grouping

regex pattern: ([a-z])(c) , Replace with : AA\2\1BB , mine is 27x faster

my regex sub 0.641000032425 s 11800000 abbAAcbBBdexfbAAceBBzeAAcxBBAAczBBzkef111anAAcnBBdehbzzdezfabbAAcbBB....

python's re sub 17.7969999313 s 11800000 abbAAcbBBdexfbAAceBBzeAAcxBBAAczBBzkef111anAAcnBBdehbzzdezfabbAAcbBB....

Cheers,
Mark

Writing Own Regular Expressions Engine for Fun

I have been rewriting my own regex in C just for fun, since I found that python's implementation is not that fast, after reading following article.

<Regular Expression Matching Can Be Simple And Fast (but is slow in Java, Perl, PHP, Python, Ruby, ...)

http://swtch.com/~rsc/regexp/regexp1.html>

I only implemented python's findall function. Actually, its relatively fast, but I am just doing it for fun.

Test string Length : 7200000 (7.2M Characters)

Sample : abbbcdexfbeczexczczkefanncdehbzzdezf * 200000

Computer : Mobile Intel Celeron 1.6MHz, 512 MB

1. a bit complicated regex, mine is 2.5x faster than python's re

regex pattern: [a]?[a-b]?[a]?[bezn][a-z0-9_]{1,3}?[e-k][^dsag]f?
my regex: 0.960999965668 s
1000000 matches [u'abbbcdexf', u'beczex', u'zczkef', u'anncdeh', u'bzzdezf', u'abbbcdexf', u'beczex'] .. [u'zczkef', u'anncde
python re: 2.3029999733 s
1000000 matches [u'abbbcdexf', u'beczex', u'zczkef', u'anncdeh', u'bzzdezf', u'abbbcdexf', u'beczex'] .. [u'zczkef', u'anncde

2. simple regex, 5x faster than python's re

regex pattern: [a-z]
my regex: 1.83299994469 s
7200000 matches [u'a', u'b', u'b', u'b', u'c', u'd', u'e'] .. [u'e', u'z', u'f']
python re: 9.35300016403 s
7200000 matches [u'a', u'b', u'b', u'b', u'c', u'd', u'e'] .. [u'e', u'z', u'f']

3. non-greedy matching, 4x faster than python's re

regex pattern: [a-z]{2,5}?
my regex: 1.40199995041 s
3600000 matches [u'ab', u'bb', u'cd', u'ex', u'fb', u'ec', u'ze'] .. [u'zz', u'de', u'zf']
python re: 5.8789999485 s
3600000 matches [u'ab', u'bb', u'cd', u'ex', u'fb', u'ec', u'ze'] .. [u'zz', u'de', u'zf']

4. fix range matching, 4x faster than python's re

regex pattern: [a-z]{5}
my regex: 0.690999984741 s
1440000 matches [u'abbbc', u'dexfb', u'eczex', u'czczk', u'efann', u'cdehb', u'zzdez'] .. [u'fannc', u'dehbz', u'zdezf']
python re: 2.66400003433 s

1440000 matches [u'abbbc', u'dexfb', u'eczex', u'czczk', u'efann', u'cdehb', u'zzdez'] .. [u'fannc', u'dehbz', u'zdezf']

Cheers,

Mark

-ize, -ise

acclimatize -ise
authorize ,-ise
amortization , -ise
agonizing,-ise
aggrandizement -ise

>>> r="(\w+iz\w*)\s*,?\s*\-(is\w*)"
>>> [re.sub(r,"\\1",INPUT),re.sub(r,lambda x:x.group(1).replace("iz","is"),INPUT]

[acclimatize,acclimatise]
[authorize,authorise]
[amortization,amortisation]
[agonizing,agonising]
[aggrandizement,aggrandisement]

I am writing dictionary parser from HTML files lately, for few weeks.

Cheers,
Soe Min