9. ソースコードのディレクトリ
•Grammar/
•Python syntax de
fi
nition in PEG
•PEP 617 -- New PEG parser for CPython
•Parser/
•Language parser (generated from PEG grammar)
•Tools/
•Supporting tools to build python including peg_generator
21. CPython の構文解析器
1. Parser/parser.c
2. 読み解こうとしない方が賢明です
3. $ wc -l parser.c
32831 parser.c
4. 人が書いたプログラムではありません
5. $ head -1 parser.c
// @generated by pegen from ./Grammar/python.gram
22. Parser Generator
•Input: Grammar/python.gram
•Output: Parser/Parser/parser.c
•Parser Generator
• Tools/peg_generator/pegen/
• This is python module.
• So, in order to build python, python is required.
• PEP 617 -- New PEG parser for CPython
• https://www.python.org/dev/peps/pep-0617/
32. struct tok_state
•Holds the state of the tokenizer
•Line buffer
•Indent stack
•etc.
•https://github.com/python/cpython/blob/
28be3191a9db2769ed05e55c6bcbccdd029656dd/
Parser/tokenizer.h#L31
33. Block de
fi
nition in PEG
block[asdl_stmt_seq*] (memo):
| NEWLINE INDENT a=statements DEDENT { a }
| simple_stmts