EVOLUTION-NINJA
Edit File: tokenizer.cpython-36.pyc
3 �b�W�G � @ s� d Z ddlmZ ddlZddlZddlZddlZddlm Z m Z m Z dddddddd�Z ddiZ dZdZd Zd Zd Zd Zd ZG dd� dejj�ZG dd� de�ZG dd� de�ZdS )z Tokenize DNS master file format� )�StringION� )�long� text_type� binary_typeT)� � � �;�(�)�"r � � � � � c @ s e Zd ZdZdS )�UngetBufferFullzDAn attempt was made to unget a token when the unget buffer was full.N)�__name__� __module__� __qualname__�__doc__� r r � /usr/lib/python3.6/tokenizer.pyr . s r c @ s� e Zd ZdZd%dd�Zdd� Zdd � Zd d � Zd d � Zdd� Z dd� Z dd� Z dd� Z dd� Z dd� Zdd� Zd d � Zd d � Zd d!� Zd"d#� Zd$S )&�Tokenz�A DNS master file format token. @ivar ttype: The token type @type ttype: int @ivar value: The token value @type value: string @ivar has_escape: Does the token value contain escapes? @type has_escape: bool � Fc C s || _ || _|| _dS )a Initialize a token instance. @param ttype: The token type @type ttype: int @param value: The token value @type value: string @param has_escape: Does the token value contain escapes? @type has_escape: bool N)�ttype�value� has_escape)�selfr r r r r r �__init__? s zToken.__init__c C s | j tkS )N)r �EOF)r r r r �is_eofM s z Token.is_eofc C s | j tkS )N)r �EOL)r r r r �is_eolP s z Token.is_eolc C s | j tkS )N)r � WHITESPACE)r r r r � is_whitespaceS s zToken.is_whitespacec C s | j tkS )N)r � IDENTIFIER)r r r r � is_identifierV s zToken.is_identifierc C s | j tkS )N)r � QUOTED_STRING)r r r r �is_quoted_stringY s zToken.is_quoted_stringc C s | j tkS )N)r �COMMENT)r r r r � is_comment\ s zToken.is_commentc C s | j tkS )N)r � DELIMITER)r r r r � is_delimiter_ s zToken.is_delimiterc C s | j tkp| j tkS )N)r r# r! )r r r r � is_eol_or_eofb s zToken.is_eol_or_eofc C s& t |t�sdS | j|jko$| j|jkS )NF)� isinstancer r r )r �otherr r r �__eq__e s z Token.__eq__c C s& t |t�sdS | j|jkp$| j|jkS )NT)r0 r r r )r r1 r r r �__ne__k s z Token.__ne__c C s d| j | jf S )Nz%d "%s")r r )r r r r �__str__q s z Token.__str__c C s | j s | S d}t| j�}d}x�||k r�| j| }|d7 }|dkr�||krPtjj�| j| }|d7 }|j� r�||krztjj�| j| }|d7 }||kr�tjj�| j| }|d7 }|j� o�|j� s�tjj�tt |�d t |�d t |� �}||7 }q W t | j |�S )Nr r r �\�d � ) r �lenr �dns� exception� UnexpectedEnd�isdigit� SyntaxError�chr�intr r )r Z unescaped�l�i�c�c2�c3r r r �unescapet s6 $ zToken.unescapec C s dS )Nr r )r r r r �__len__� s z Token.__len__c C s t | j| jf�S )N)�iterr r )r r r r �__iter__� s zToken.__iter__c C s$ |dkr| j S |dkr | jS t�d S )Nr r )r r � IndexError)r rA r r r � __getitem__� s zToken.__getitem__N)r F)r r r r r r" r$ r& r( r* r, r. r/ r2 r3 r4 rE rF rH rJ r r r r r 3 s" r c @ s� e Zd ZdZejdfdd�Zdd� Zdd� Zd d � Z d d � Z d(dd�Z dd� Z dd� Z e Zdd� Zdd� Zdd� Zdd� Zd d � Zd)d d �Zd*d d!�Zd+d"d#�Zd$d%� Zd&d'� ZdS ),� Tokenizera� A DNS master file format tokenizer. A token is a (type, value) tuple, where I{type} is an int, and I{value} is a string. The valid types are EOF, EOL, WHITESPACE, IDENTIFIER, QUOTED_STRING, COMMENT, and DELIMITER. @ivar file: The file to tokenize @type file: file @ivar ungotten_char: The most recently ungotten character, or None. @type ungotten_char: string @ivar ungotten_token: The most recently ungotten token, or None. @type ungotten_token: (int, string) token tuple @ivar multiline: The current multiline level. This value is increased by one every time a '(' delimiter is read, and decreased by one every time a ')' delimiter is read. @type multiline: int @ivar quoting: This variable is true if the tokenizer is currently reading a quoted string. @type quoting: bool @ivar eof: This variable is true if the tokenizer has encountered EOF. @type eof: bool @ivar delimiters: The current delimiter dictionary. @type delimiters: dict @ivar line_number: The current line number @type line_number: int @ivar filename: A filename that will be returned by the L{where} method. @type filename: string Nc C s� t |t�r t|�}|dkr`d}n@t |t�rDt|j� �}|dkr`d}n |dkr`|tjkr\d}nd}|| _d| _d| _ d| _ d| _ d| _ t | _d| _|| _dS )a� Initialize a tokenizer instance. @param f: The file to tokenize. The default is sys.stdin. This parameter may also be a string, in which case the tokenizer will take its input from the contents of the string. @type f: file or string @param filename: the name of the filename that the L{where} method will return. @type filename: string Nz<string>z<stdin>z<file>r Fr )r0 r r r �decode�sys�stdin�file� ungotten_char�ungotten_token� multiline�quoting�eof� _DELIMITERS� delimiters� line_number�filename)r �frX r r r r � s* zTokenizer.__init__c C sZ | j dkrJ| jrd}qV| jjd�}|dkr2d| _qV|dkrV| jd7 _n | j }d| _ |S )z<Read a character from input. @rtype: string Nr r Tr )rP rT rO �readrW )r rB r r r � _get_char� s zTokenizer._get_charc C s | j | jfS )z�Return the current location in the input. @rtype: (string, int) tuple. The first item is the filename of the input, the second is the current line number. )rX rW )r r r r �where� s zTokenizer.wherec C s | j dk rt�|| _ dS )aE Unget a character. The unget buffer for characters is only one character large; it is an error to try to unget a character when the unget buffer is not empty. @param c: the character to unget @type c: string @raises UngetBufferFull: there is already an ungotten char N)rP r )r rB r r r � _unget_char s zTokenizer._unget_charc C sL d}xB| j � }|dkr<|dkr<|dks.| j r<| j|� |S |d7 }qW dS )a( Consume input until a non-whitespace character is encountered. The non-whitespace character is then ungotten, and the number of whitespace characters consumed is returned. If the tokenizer is in multiline mode, then newlines are whitespace. @rtype: int r r r r r N)r[ rR r] )r �skippedrB r r r �skip_whitespace s zTokenizer.skip_whitespaceFc C sT | j dk r>| j }d| _ |j� r(|r>|S n|j� r:|r>|S n|S | j� }|r\|dkr\ttd�S d}t}d}�x�| j� }|dks�|| jk�r|dkr�| j r�t j j �|dko�|t k�r|dkr�| jd7 _| j� ql�q |dk�r | jdkr�t j j�| jd8 _| j� qln�|d k�rH| j �s0d | _ t| _t }qlnd| _ t| _| j� qln�|d k�r\ttd �S |d k�r�x,| j� }|d k�s�|dk�r�P ||7 }�qhW |�r�| j|� tt|�S |dk�r�| j�r�t j jd ��tt�S | j�r�| j� d}qln ttd �S n|}t}n | j|� P � n�| j �r�|dk�r�| j� }|dk�r>t j j �|j� �r�| j� }|dk�rbt j j �| j� } |dk�r|t j j �|j� �o�| j� �s�t j j�tt|�d t|�d t| � �}n|d k�rt j jd��n:|dk�r||7 }d }| j� }|dk�s|d k�rt j j �||7 }qlW |dk�rH|t k�rH| j�rDt j jd ��t}t|||�S )a Get the next token. @param want_leading: If True, return a WHITESPACE token if the first character read is whitespace. The default is False. @type want_leading: bool @param want_comment: If True, return a COMMENT token if the first token read is a comment. The default is False. @type want_comment: bool @rtype: Token object @raises dns.exception.UnexpectedEnd: input ended prematurely @raises dns.exception.SyntaxError: input was badly formed Nr r r Fr r r r Tr r zunbalanced parenthesesr5 r6 r7 znewline in quoted string)rQ r& r, r_ r r% r' r[ rV rS r9 r: r; r) rR r= �_QUOTING_DELIMITERSrU r# r] r+ r! r- r< r> r? ) r Z want_leadingZ want_comment�tokenr^ r r rB rC rD r r r �get% s� & z Tokenizer.getc C s | j dk rt�|| _ dS )a@ Unget a token. The unget buffer for tokens is only one token large; it is an error to try to unget a token when the unget buffer is not empty. @param token: the token to unget @type token: Token object @raises UngetBufferFull: there is already an ungotten token N)rQ r )r ra r r r �unget� s zTokenizer.ungetc C s | j � }|j� rt�|S )zLReturn the next item in an iteration. @rtype: (int, string) )rb r" � StopIteration)r ra r r r �next� s zTokenizer.nextc C s | S )Nr )r r r r rH � s zTokenizer.__iter__c C s@ | j � j� }|j� s tjjd��|jj� s6tjjd��t|j�S )z|Read the next token and interpret it as an integer. @raises dns.exception.SyntaxError: @rtype: int zexpecting an identifierzexpecting an integer) rb rE r( r9 r: r= r r< r? )r ra r r r �get_int� s zTokenizer.get_intc C s, | j � }|dk s|dkr(tjjd| ��|S )z�Read the next token and interpret it as an 8-bit unsigned integer. @raises dns.exception.SyntaxError: @rtype: int r � z#%d is not an unsigned 8-bit integer)rf r9 r: r= )r r r r r � get_uint8� s zTokenizer.get_uint8c C s, | j � }|dk s|dkr(tjjd| ��|S )z�Read the next token and interpret it as a 16-bit unsigned integer. @raises dns.exception.SyntaxError: @rtype: int r i�� z$%d is not an unsigned 16-bit integer)rf r9 r: r= )r r r r r � get_uint16� s zTokenizer.get_uint16c C sh | j � j� }|j� s tjjd��|jj� s6tjjd��t|j�}|dk sT|td�krdtjjd| ��|S )z�Read the next token and interpret it as a 32-bit unsigned integer. @raises dns.exception.SyntaxError: @rtype: int zexpecting an identifierzexpecting an integerr l z$%d is not an unsigned 32-bit integer) rb rE r( r9 r: r= r r< r )r ra r r r r � get_uint32� s zTokenizer.get_uint32c C s. | j � j� }|j� p|j� s(tjjd��|jS )z}Read the next token and interpret it as a string. @raises dns.exception.SyntaxError: @rtype: string zexpecting a string)rb rE r( r* r9 r: r= r )r �originra r r r � get_string� s zTokenizer.get_stringc C s& | j � j� }|j� s tjjd��|jS )z�Read the next token and raise an exception if it is not an identifier. @raises dns.exception.SyntaxError: @rtype: string zexpecting an identifier)rb rE r( r9 r: r= r )r rk ra r r r �get_identifier s zTokenizer.get_identifierc C s, | j � }|j� s tjjd��tjj|j|�S )z�Read the next token and interpret it as a DNS name. @raises dns.exception.SyntaxError: @rtype: dns.name.Name objectzexpecting an identifier)rb r( r9 r: r= �name� from_textr )r rk ra r r r �get_name s zTokenizer.get_namec C s. | j � }|j� s(tjjd|j|jf ��|jS )z�Read the next token and raise an exception if it isn't EOL or EOF. @raises dns.exception.SyntaxError: @rtype: string z expected EOL or EOF, got %d "%s")rb r/ r9 r: r= r r )r ra r r r �get_eol! s zTokenizer.get_eolc C s. | j � j� }|j� s tjjd��tjj|j�S )Nzexpecting an identifier) rb rE r( r9 r: r= Zttlro r )r ra r r r �get_ttl0 s zTokenizer.get_ttl)FF)N)N)N)r r r r rM rN r r[ r\ r] r_ rb rc re �__next__rH rf rh ri rj rl rm rp rq rr r r r r rK � s( $ { rK )r �ior rM Z dns.exceptionr9 Zdns.nameZdns.ttlZ_compatr r r rU r` r! r# r% r' r) r+ r- r: Z DNSExceptionr �objectr rK r r r r �<module> s0 o