^V{1@sddlmZmZmZddlmZddlmZddlZddl Z ddl m Z m Z m Z mZddl mZmZddlmZdd lmZydd lmZWnek reZYnXydd lmZWn(ek rGd d d eZYnXedde DZedde DZeddeDZeeddgBZdZejre j ee!dZ"ne j eZ"e#ddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5g Z$e j d6Z%iZ&Gd7d8d8eZ'dd9d9d:d;Z(Gd<d=d=eZ)Gd>d?d?e)Z*Gd@dAdAe+Z,GdBdCdCeZ-GdDdEdEeZ.dFdGZ/dS)H)absolute_importdivisionunicode_literals) text_type) http_clientN)EOFspaceCharacters asciiLettersasciiUppercase) encodingsReparseException)utils)StringIO)BytesIO)BufferedIOBasec@seZdZdS)rN)__name__ __module__ __qualname__rr?/tmp/pip-build-9m6vxulb/pip/pip/_vendor/html5lib/inputstream.pyrs rcCsg|]}|jdqS)ascii)encode).0itemrrr s rcCsg|]}|jdqS)r)r)rrrrrrs cCsg|]}|jdqS)r)r)rrrrrrs >t|j||krd|t|j|8}|d7}q'W||g|_dS)Nrr)_bufferedBytesAssertionErrorr%r!r")r#r&offsetirrrseekMszBufferedStream.seekcCsp|js|j|S|jdt|jkr_|jdt|jdkr_|j|S|j|SdS)Nrrr)r! _readStreamr"r%_readFromBuffer)r#bytesrrrreadVs     zBufferedStream.readcCstdd|jDS)NcSsg|]}t|qSr)r%)rrrrrr`s z1BufferedStream._bufferedBytes..)sumr!)r#rrrr)_szBufferedStream._bufferedBytescCsJ|jj|}|jj||jdd7 Normalized stream from source for use by html5lib. source can be either a file-object, local filename or a string. The optional encoding parameter must be a string that indicates the encoding. If specified, that encoding will be used, regardless of any BOM or later declaration (such as in a meta element) parseMeta - Look for a element containing encoding information Nu􏿿rz"[\uD800-\uDFFF]"zJ"([\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?.z^%sz[%s]+N)charsUntilRegExKeyErrorror*r6rPrQr~r'r[rZendr3rj) r# charactersoppositecharsrregexr8mrrrrr charsUntilFs2   &    z!HTMLUnicodeInputStream.charsUntilcCso|dk rk|jdkr=||j|_|jd7_qk|jd8_|j|j|ksktndS)Nrr)r[r'rZr*)r#rkrrrungetus  zHTMLUnicodeInputStream.unget)rrrr=rnr$rXrVrgr"rkrjrOrSrrrrrrrBs  &    )  /rBc@syeZdZdZdddddZddZdd Zddd d Zd d ZddZ ddZ dS)rCzProvides a unicode stream of characters to the HTMLTokenizer. This class takes care of character encoding and removing or replacing incorrect byte-sequences and also provides column and line tracking. NTcCs|j||_tj||jt|df|_d|_d|_d|_|jddkr|j |||_n|j dS)aInitialises the HTMLInputStream. HTMLInputStream(source, [encoding]) -> Normalized stream from source for use by html5lib. source can be either a file-object, local filename or a string. The optional encoding parameter must be a string that indicates the encoding. If specified, that encoding will be used, regardless of any BOM or later declaration (such as in a meta element) parseMeta - Look for a element containing encoding information rKidz windows-1252rN) rV rawStreamrBr$ codecNamerU numBytesMetanumBytesChardetdefaultEncodingdetectEncodingrX)r#rDrErFrGrrrr$s   zHTMLBinaryInputStream.__init__cCs6tj|jd|jd|_tj|dS)Nrrq)codecs getreaderrUrrWrBrX)r#rrrrXs zHTMLBinaryInputStream.resetc CsVt|dr|}n t|}y|j|jWnt|}YnX|S)zvProduces a file object from source. source can be either a file object, local filename or a string. r1)r@rr-r(r)r#rDr rrrrVs  z HTMLBinaryInputStream.openStreamc Cs|j}d}|dkr9|r9|j}d}n|dkr:|r:d}yyddlm}Wn"tk rddlm}YnXg}|}x[|js|jj|j }t |t st |sPn|j ||j|qW|j|jd}|jjdWq:tk r6Yq:Xn|dkrXd}|j}nidd6} |j| kr| |j}n||fS)NrK tentativer)UniversalDetectorrEz windows-1252z iso-8859-1) detectBOMdetectEncodingMetacharade.universaldetectorr ImportErrorchardet.universaldetectordonerr1rr>r0r*r3feedcloseresultr-rlower) r#rFrGrE confidencerbuffersdetectorr! encodingSubrrrrsB             z$HTMLBinaryInputStream.detectEncodingcCs|jddkstt|}|d kr:d}n|dkrJdS||jdkrv|jddf|_nF|jjd|j|df|_td|jd|fdS) NrrKutf-16 utf-16-be utf-16-lezutf-8rzEncoding changed from %s to %s)rrr)rUr*rrr-rXr )r# newEncodingrrrchangeEncodings     z$HTMLBinaryInputStream.changeEncodingcCsidtj6dtj6dtj6dtj6dtj6}|jjd}t|t s_t |j |dd}d}|s|j |}d}|s|j |dd }d }qn|jj |r|pd |S) zAttempts to detect at BOM at the start of the stream. If an encoding can be determined from the BOM return the name of the encoding otherwise return Nonezutf-8z utf-16-lez utf-16-bez utf-32-lez utf-32-beNrvr) rBOM_UTF8 BOM_UTF16_LE BOM_UTF16_BE BOM_UTF32_LE BOM_UTF32_BErr1r>r0r*getr-)r#bomDictstringrEr-rrrr s   zHTMLBinaryInputStream.detectBOMcCsk|jj|j}t|ts*tt|}|jjd|j}|dkrgd}n|S)z9Report the encoding declared by the meta element rutf-16 utf-16-be utf-16-lezutf-8)rrr) rr1rr>r0r*EncodingParserr- getEncoding)r#r!parserrErrrr)s    z(HTMLBinaryInputStream.detectEncodingMeta) rrrr=r$rXrVrrrrrrrrrCs (  -  rCc@seZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ e e e Z ddZe eZeddZddZddZddZdS) EncodingByteszString-like object with an associated position and various extra methods If the position is ever greater than the string length then an exception is raisedcCs+t|tsttj||jS)N)r>r0r*__new__r)r#valuerrrr<szEncodingBytes.__new__cCs d|_dS)Nrr)rg)r#rrrrr$@szEncodingBytes.__init__cCs|S)Nr)r#rrr__iter__CszEncodingBytes.__iter__cCsV|jd}|_|t|kr/tn|dkrDtn|||dS)Nrr)rgr% StopIterationrA)r#prrr__next__Fs    zEncodingBytes.__next__cCs |jS)N)r)r#rrrnextNszEncodingBytes.nextcCs\|j}|t|kr$tn|dkr9tn|d|_}|||dS)Nrr)rgr%rrA)r#rrrrpreviousRs    zEncodingBytes.previouscCs+|jt|krtn||_dS)N)rgr%r)r#r"rrr setPosition[s zEncodingBytes.setPositioncCs<|jt|krtn|jdkr4|jSdSdS)Nr)rgr%r)r#rrr getPosition`s  zEncodingBytes.getPositioncCs||j|jdS)Nr)r")r#rrrgetCurrentBytejszEncodingBytes.getCurrentBytecCsf|j}xM|t|krX|||d}||krK||_|S|d7}q W||_dS)zSkip past a list of charactersrN)r"r%rg)r#rrrrrrr}os    zEncodingBytes.skipcCsf|j}xM|t|krX|||d}||krK||_|S|d7}q W||_dS)Nr)r"r%rg)r#rrrrrr skipUntil{s    zEncodingBytes.skipUntilcCsT|j}|||t|}|j|}|rP|jt|7_n|S)zLook for a sequence of bytes at the start of a string. If the bytes are found return True and advance the position to the byte after the match. Otherwise return False and leave the position alone)r"r% startswith)r#r0rr4r8rrr matchBytess  zEncodingBytes.matchBytescCsn||jdj|}|dkrd|jdkrCd|_n|j|t|d7_dStdS)zLook for the next sequence of bytes matching a given sequence. If a match is found advance the position to the last byte of the matchNrrTrr)r"findrgr%r)r#r0 newPositionrrrjumpTos  zEncodingBytes.jumpToN)rrrr=rr$rrrrrrpropertyr"r currentBytespaceCharactersBytesr}rrrrrrrr8s           rc@seZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ dS)rz?Mini parser for detecting character encoding from meta elementscCst||_d|_dS)z3string - the data to work on for encoding detectionN)rr4rE)r#r4rrrr$szEncodingParser.__init__c Csd|jfd|jfd|jfd|jfd|jfd|jff}xw|jD]l}d}xS|D]K\}}|jj|rky|}PWqtk rd}PYqXqkqkW|sXPqXqXW|jS) Ns)r4r)r#rrrrszEncodingParser.handleCommentcCsE|jjtkrdSd}d}x|j}|dkrAdS|ddkr|ddk}|r=|dk r=||_dSq%|ddkr|d}t|}|dk r=||_dSq%|ddkr%tt|d}|j}|dk r=t|}|dk r:|r.||_dS|}q:q=q%q%WdS) NTFrs http-equivrs content-typescharsetscontent) r4rr getAttributerErContentAttrParserrparse)r# hasPragmapendingEncodingattrtentativeEncodingcodec contentParserrrrrs:            zEncodingParser.handleMetacCs |jdS)NF)handlePossibleTag)r#rrrrsz%EncodingParser.handlePossibleStartTagcCst|j|jdS)NT)rr4r)r#rrrrs z#EncodingParser.handlePossibleEndTagcCs|j}|jtkr9|r5|j|jndS|jt}|dkra|jn+|j}x|dk r|j}qpWdS)NTr)r4rasciiLettersBytesrrrspacesAngleBracketsr)r#endTagr4rrrrrrs      z EncodingParser.handlePossibleTagcCs|jjdS)Nr)r4r)r#rrrrszEncodingParser.handleOthercCs|j}|jttdgB}|dksIt|dksIt|d krYdSg}g}x|dkr~|r~Pnz|tkr|j}Pn^|d krdj|dfS|tkr|j|j n|dkrdS|j|t |}qhW|dkr1|j dj|dfSt ||j}|d kr|}xt |}||krt |dj|dj|fS|tkr|j|j q\|j|q\Wn^|dkrdj|dfS|tkr|j|j n|dkr!dS|j|xwt |}|t kredj|dj|fS|tkr|j|j q1|dkrdS|j|q1WdS) z_Return a name,value pair for the next attribute in the stream, if one is found, or None/Nrr=r5'")rN)rr)rr) r4r}r frozensetr%r*r6asciiUppercaseBytesr3rrrr)r#r4rattrName attrValue quoteCharrrrrsh $                        zEncodingParser.getAttributeN) rrrr=r$rrrrrrrrrrrrrs     $    rc@s(eZdZddZddZdS)rcCs"t|tst||_dS)N)r>r0r*r4)r#r4rrrr$UszContentAttrParser.__init__cCsNy1|jjd|jjd7_|jj|jjdksHdS|jjd7_|jj|jjdkr|jj}|jjd7_|jj}|jj|r|j||jjSdSn]|jj}y+|jjt|j||jjSWn#tk r/|j|dSYnXWntk rIdSYnXdS)Nscharsetrrrr)rr)r4rr"r}rrrr)r# quoteMark oldPositionrrrrYs.       zContentAttrParser.parseN)rrrr$rrrrrrTs  rc Cstt|tr>y|jd}Wq>tk r:dSYq>Xn|rltjd|j}tj|dSdSdS)z{Return the python codec name corresponding to an encoding or None if the string doesn't correspond to a valid encoding.rNrY) r>r0decodeUnicodeDecodeErrorascii_punctuation_rerprr r)rE canonicalNamerrrr{s  r)0 __future__rrrpip._vendor.sixrpip._vendor.six.movesrrrP constantsrr r r r r rYriorrrrobjectrrrrrinvalid_unicode_no_surrogaterLrQrRrtsetr|rrrrIrBrCr0rrrrrrrrsT  "     Jg'