Skip to content

baron raise exception when unicode is in code  #175

@James4Ever0

Description

@James4Ever0

example:

the unicode below are just attributes inside a class.

orders=[
        BSP.all.order.最多点击,
        BSP.all.order.最多收藏,
        BSP.all.order.最新发布,
        BSP.all.order.最多弹幕,
        BSP.all.order.综合排序,
    ]

exception:

Traceback (most recent call last):
  File "/media/root/parrot/pyjom/tests/skipexception_code_and_continue_resurrection_time_travel_debugging/codemod_redbaron/redbaron_test.py", line 6, in <module>
    r=redbaron.RedBaron(t)
  File "/usr/local/lib/python3.9/dist-packages/redbaron/redbaron.py", line 36, in __init__
    self.node_list = base_nodes.NodeList.from_fst(baron.parse(source_code), parent=self, on_attribute="root")
  File "/usr/local/lib/python3.9/dist-packages/baron/baron.py", line 49, in parse
    tokens = tokenize(source_code, False)
  File "/usr/local/lib/python3.9/dist-packages/baron/baron.py", line 70, in tokenize
    splitted = split(pouet)
  File "/usr/local/lib/python3.9/dist-packages/baron/spliter.py", line 6, in split
    return list(split_generator(sequence))
  File "/usr/local/lib/python3.9/dist-packages/baron/spliter.py", line 76, in split_generator
    raise UntreatedError("Untreated elements: %s" % iterator.rest_of_the_sequence().__repr__()[:50])
baron.spliter.UntreatedError: Untreated elements: '最多点击,\n        BSP.all.order.最多收藏,\n        BSP.a

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions