Memo# - Regular Expressions - Part 2 - Substitutions with Grouping

Test String

7800000 (7.8M Characters)

"abbbcdexfbeczexczczkef111anncdehbzzdezf" * 200000

Very Simple Replace

regex pattern: bc , Replace with : TEST , mine is 2.5x faster

my regex sub 0.140000104904 s 8200000 abbTESTdexfbeczexczczkef111anncdehbzzdezfabbTESTdexfbeczexczczkef111anncde....

python's re sub 0.359999895096 s 8200000 abbTESTdexfbeczexczczkef111anncdehbzzdezfabbTESTdexfbeczexczczkef111anncde....

With Range Matching

regex pattern: [a-z]c , Replace with : TEST , mine is 6x faster

my regex sub 0.453000068665 s 9800000 abbTESTdexfbTESTzeTESTTESTzkef111anTESTdehbzzdezfabbTESTdexfbTESTze....

python's re sub 2.82800006866 s 9800000 abbTESTdexfbTESTzeTESTTESTzkef111anTESTdehbzzdezfabbTESTdexfbTESTze....

With Range Matching + Grouping

regex pattern: ([a-z])(c) , Replace with : AA\2\1BB , mine is 27x faster

my regex sub 0.641000032425 s 11800000 abbAAcbBBdexfbAAceBBzeAAcxBBAAczBBzkef111anAAcnBBdehbzzdezfabbAAcbBB....

python's re sub 17.7969999313 s 11800000 abbAAcbBBdexfbAAceBBzeAAcxBBAAczBBzkef111anAAcnBBdehbzzdezfabbAAcbBB....

Cheers,
Mark

Writing Own Regular Expressions Engine for Fun

I have been rewriting my own regex in C just for fun, since I found that python's implementation is not that fast, after reading following article.

<Regular Expression Matching Can Be Simple And Fast (but is slow in Java, Perl, PHP, Python, Ruby, ...)

http://swtch.com/~rsc/regexp/regexp1.html>

I only implemented python's findall function. Actually, its relatively fast, but I am just doing it for fun.

Test string Length : 7200000 (7.2M Characters)

Sample : abbbcdexfbeczexczczkefanncdehbzzdezf * 200000

Computer : Mobile Intel Celeron 1.6MHz, 512 MB

1. a bit complicated regex, mine is 2.5x faster than python's re

regex pattern: [a]?[a-b]?[a]?[bezn][a-z0-9_]{1,3}?[e-k][^dsag]f?
my regex: 0.960999965668 s
1000000 matches [u'abbbcdexf', u'beczex', u'zczkef', u'anncdeh', u'bzzdezf', u'abbbcdexf', u'beczex'] .. [u'zczkef', u'anncde
python re: 2.3029999733 s
1000000 matches [u'abbbcdexf', u'beczex', u'zczkef', u'anncdeh', u'bzzdezf', u'abbbcdexf', u'beczex'] .. [u'zczkef', u'anncde

2. simple regex, 5x faster than python's re

regex pattern: [a-z]
my regex: 1.83299994469 s
7200000 matches [u'a', u'b', u'b', u'b', u'c', u'd', u'e'] .. [u'e', u'z', u'f']
python re: 9.35300016403 s
7200000 matches [u'a', u'b', u'b', u'b', u'c', u'd', u'e'] .. [u'e', u'z', u'f']

3. non-greedy matching, 4x faster than python's re

regex pattern: [a-z]{2,5}?
my regex: 1.40199995041 s
3600000 matches [u'ab', u'bb', u'cd', u'ex', u'fb', u'ec', u'ze'] .. [u'zz', u'de', u'zf']
python re: 5.8789999485 s
3600000 matches [u'ab', u'bb', u'cd', u'ex', u'fb', u'ec', u'ze'] .. [u'zz', u'de', u'zf']

4. fix range matching, 4x faster than python's re

regex pattern: [a-z]{5}
my regex: 0.690999984741 s
1440000 matches [u'abbbc', u'dexfb', u'eczex', u'czczk', u'efann', u'cdehb', u'zzdez'] .. [u'fannc', u'dehbz', u'zdezf']
python re: 2.66400003433 s

1440000 matches [u'abbbc', u'dexfb', u'eczex', u'czczk', u'efann', u'cdehb', u'zzdez'] .. [u'fannc', u'dehbz', u'zdezf']

Cheers,

Mark

ေတာှေနးဒိုး - Tornado - Web Server written in Python

http://www.tornadoweb.org/

Facebook ရဲ့ FriendConnect ဆိုတဲ့ ေနရာမွာ အသံုးျပုထားတယ္လို့ဆိုပါတယ္။ တစ္စကဿကန့္ကို request ေထာင္နဲ့ခ်ီျပီး အလုပ္လုပ္နိုင္တယ္လို့ဆိုပါတယ္။

ထြက္ျပီး တစ္ရက္အျကာ မွာ http://blog.shibu.jp/article/32075270.html ဂ်ပန္ဘာသာကို ဘာသာျပန္တဲ့သူေတြက ျပန္ျပီးသြားပါျပီ။ :-)

nginx ကို ဆာဗာေရွ့မွာခံျပီး အလုပ္လုပ္ရင္ ပိုျမန္ပံုရပါတယ္။ က်ေနာှေတာ့ Search Engine မွာ သံုးဖို့ စာရင္း List ထဲ ထည့္ထားလိုက္ျပီ။

မာ့ခ္

#Memo - Mecab - Japanese Part-Of-Speech and Morphological Analyzer

#Memo - Mecab - Japanese Part-Of-Speech and Morphological Analyzer

http://mecab.sourceforge.net

# I really like to write one for Myanmar too. Still needs some knowledges.

Untitled!

Yay! Almost success Zawgyi to Unicode 5.1 and Unicode 5.1 to Zawgyi Converter in Pure C with 60MB+/s

Zawgyi Normalization Program

normalize.7z

- extract and run normalize.exe test.txt 6 in command line (Command Prompt).
(6 is zawgyi for this exe file)
- Input file must be UTF8 encoded text file
- outfile will be test.out

Able to fix the following things

ေမာင္ ္ ္ ္ ္ က ု       ိ  က ိ      ု ၏၏၏၏၏၏၏။။။။။။။။။။။။။။။။။။။
ျမျကျခ  ျကျကိ ျကြ ျကြိ ျခ ျခြ ျခိ ျခြိ ျခ  ျခြ  ျခိ   ျခြီ

and result will be like this.

ေမာင္္ ကို ကို၏။
ျမျကျချကျကိျကြျကြိျချခြျခိျခြိျချခြျခိျခြီ

Here is my license

- Cannot be used in Commercial related programs, applications, anything , related with commercial things
- Must be credit to me, if you use and put your data or program online.

Cheers,
Soe Min

Burglish Update v1.9.2

updates for unicode 5.1 direct input layout (typewriter input)

In Unicode 5.1, သေဝထိုး and ရရစ္ need to store after consonent(က-အ), like this က, ေက

But from user side, most people want to type ဩေဝထိုး first.

so, i reorder the input sequence after you type in burglish test area.

http://burglish.googlepages.com/testarea.htm
or
http://burglish.my-mm.org/latest/trunk/web/testarea.htm

when you type ေက, it will automatically fix က, and ေျက to က , if you have proper unicode 5.1 font(Myanmar3,Padauk,Parabaik) and render engine (eg. latest uniscribe), you will not see like this ျက , you will instead see, ျက

for ေျမာင္, you need to type ေမ ာ ျင ္. it will properly convert to unicode 5.1 compatible one မ ာ ေျင ္.

and few others update.

Zawgyi's some useful shortcuts like,

Shift Q -်ွ
Shift W -်ြွ
Shift E - ်ြ
Shift K -ွု

can also type in Unicode 5.1 (Myanmar3) through Burglish Test Engine

Cheers,
Soe Min

Regular Verb မွ Simple Past ေျပာင္းပံုေျပာင္းနည္း (Python)

Original Article here.
http://web2.uvcs.uvic.ca/elc/StudyZone/330/grammar/pasted.htm (thx to Ma TTA)

rule 1
live, date တို့လို e နဲ့ဆံုးတာေတြကို "d" ပဲေပါင္းေပး lived, dated

>>> if word[-1]=="e":word+"d"

rule 2
try, cry တို့လို y နဲ့ဆံုးတာေတြကို ied ေျပာင္း tried, cried

>>> if word[-1]=="y":word[:-1]+"ied"

rule 3 (နဲနဲရွုပ္တယ္)
tap, commit တို့လို့စာလံုးေတြကို က်ေတာ့ ေနာက္ဆံုးစာလံုးကို ၂လံုးလုပ္
ေနာက္ဆံုးကေန သံုးလံုးမွာ ဗ်ည္း၊ သရ၊ ဗ်ည္းဆိုျပီး သြားေနရင္ rule 3 နဲ့ ညိတယ္လို့ ယူဆပါတယ္။

>>> if word[-3] not in "aeiuo" and word[-2] in "aeiuo" and word[-1] not in "aeiuowy": word+word[-1]+"ed"

rule 4
အထက္ကဟာေတြ နဲ့မညိတာေတြ အားလံုးကို ed ေပါင္းေပး

>>> word+"ed"

cheers,
Soe Min

Morphy

% python
>>> from wordnet import *
>>> morphy("ate",VERB)
eat
>>> morphy("easier",ADJ)
easy
>>> morphy("apples")
apple

nice huh!

Search POS (Part of Speech) using NLTK

% python
>>> import nltk
>>> nltk.pos_tag(["work"])
[('work', 'NN')]

Oh! its say work as NOUN!

Yeah, single verb can be use as noun. we need to pass the the whole sentence to check it

we have english sample like "he works for a company"

>>> nltk.pos_tag("he works for a company".split())
[('he', 'PRP'), ('works', 'VBZ'), ('for', 'IN'), ('a', 'DT'), ('company', 'NN')]

wow :-) its show "works" as VBZ, it is verb, great. huh!

so search, "work" in nltk results again. and can get very accurate POS :-)

work = VBZ for that case :-)

#but unfortunately its take time around 100ms to 1s, its slow for me, but no way. have to use that.

cheers,
soe min

Researching Grammar of Japanese and Burmese for (machine?) translation... 1

やる　လုပျ
やらない မလုပျဘူး
やります လုပျမညျ
やりません မလုပျဘူး
やった လုပျခဲ့
やりました လုပျခဲ့
やらなかった မလုပျခဲ့
やりませんでした မလုပျခဲ့ဘူး
やらず မလုပျပဲ
やらぬ မလုပျ
やらざるをえない မလုပျလို့ မရ
やっている လုပျန
やっています လုေပျနသေညျ
やっていて လုပျနပွီးေ
やっていた　လုပျနခဲ့ေ
やっていました　လုပျနခဲ့ေသညျ
やらないと မလုပျရငျ (မလယျဘူး)
やらなきゃ မလုပျလို့ (မရ)
やろう　လုပျရအာငေျ
やりましょう　လုပျရအာငေျ
やれる　လုပျနိုငျ
やられる　လုပျခံရ (ကိုယျက)
やらされる　လုပျခိုျငးခံရ (ကိုယျက)
やらす　လုပျခိုျငး (သူမားကြို)
やらせる လုပျခိုျငး (သူမားကြို)
やらさせる လုပျခိုျငး (သူမားကြို)
やらさせられる လုပျခိုျငးခံရ (သူမားကြို)
やって လုပျပွီး (so many patterns)
やっても လုပျရငျတာငေျ
やったから လုပျရငျ (လုပျခဲ့တာကွာင့ေျ)
やってから လုပျပွီးမ
やって... etc...

.......