A python module for reliq library.
pip install reliq
from reliq import reliq
from reliq import reliq
html = ""
with open('index.html','r') as f:
html = f.read()
rq = reliq(html) #parse html
expr = reliq.expr(r"""
div .user; {
a href; {
.name @ | "%i",
.link @ | "%(href)v"
},
.score.u span .score,
.info dl; {
.key dt | "%i",
.value dd | "%i"
} |,
.achievements.a li class=b>"achievement-" | "%i\n"
}
""") #expressions can be compiled
users = []
links = []
for i in rq.filter(r'table; { tr, text@ iw>lisp }')[:-2]:
# ignore comments and text nodes
if i.type is not reliq.Type.tag:
continue
first_child = i[0]
if first_child.child_count < 3 and first_child.name == "div" and first_child.starttag == '<div>':
continue
link = first_child[2].attribs['href']
if re.match('^https://$',link):
links.append(link)
continue
#make sure that object is an ancestor of <main> tag
for j in i.ancestors():
if j.name == "main":
break
else:
continue
#search() returns str, in this case expression is already compiled
# but can be also passed as a str() or bytes(). If Path() is passed
# file will be read
user = json.loads(i.search(expr))
users.append(user)
try: #handle errors
reliq.search('p / /','<p></p>')
except reliq.ScriptError: # all errors inherit from reliq.Error
print("error")
#get text from all text nodes that are descendants of object
print(rq[2].text_recursive)
#get text from all text nodes that are children of object
print(rq[2].text)
#decode html entities
reliq.decode('loop & < ⃛ Ô')
#convert to json
rq.json(r"""
.files * #files; ( li )( span .head ); {
.type i class child@ | "%(class)v" / sed "s/^flaticon-//",
.name @ | "%Dt" / trim sed "s/ ([^)]* [a-zA-Z][Bb])$//",
.size @ | "%t" / sed 's/.* \(([^)]* [a-zA-Z][Bb])\)$/\1/; s/,//g; /^[0-9].* [a-zA-Z][bB]$/!d' "E"
} |
""") #json format is not enforced, so incorrect script will raise exceptions from json.loads()
Everything is contained inside one class
from reliq import reliq
reliq object takes a single argument representing html, this can be str(), bytes(), Path() (file is read as bytes), reliq() or None.
rq = reliq('<p>Example</p>') #passed directly
rq2 = reliq(Path('index.html')) #passed from file
rq3 = reliq(None) # empty object
rq4 = reliq() # empty object
reliq can have 5 types that change the behaviour of methods.
Calling type property on object e.g. rq.type returns instance of reliq.Type(Flag).
Gets returned from either reliq(None) or reliq.filter() that matches nothing, makes all methods return default values.
Similar to empty but should never happen
Returned by successful initialization e.g.
reliq('<p>Example</p>')
Returned by reliq.filter() that succeeds
Returned by axis methods or by accessing the object like a list.
The type itself is a grouping of more specific types:
Returns the same html from which the object was compiled.
If first argument is True or raw=True returns bytes.
data = Path('index.html').read_bytes
rq = reliq(data)
x = rq[0][2][1][8]
# if both objects are bytes() then their ids should be the same
x.get_data(True) is data
Full string representation of current object
rq = reliq("""
<h1><b>H</b>1</h1>
<h2>N2</h2>
<h2>N3</h2>
""")
str(rq) # struct
# '\n <h1><b>H</b>1</h1>\n <h2>N2</h2>\n <h2>N3</h2>\n'
str(rq.filter('h2')) # list
# '<h2>N2</h2><h2>N3</h2>'
str(rq[0]) # single
# '<h1><b>H</b>1</h1>'
str(reliq()) # empty
# ''
For single indexes results from children() axis, otherwise from self() axis
rq = reliq('<div><p>1</p> Text <b>H</b></div>')
first = rq[0] # struct
# <div>
first[1] # single
# <b>
r = first.filter('( text@ * )( * ) child@')
r[1] # list
# " Text " obj
r[2] == first[1]
Amount of objects returned from __getitem__
Calling these properties for types other than single returns their default values.
lvl -> int level in html structure
rlvl -> int level in html structure, relative to parent
position -> int position in html structure
rposition -> int position in html structure, relative to parent
Calling some properties makes sense only for certain types.
tag_count -> int count of tags
text_count -> int count of text
comment_count -> int count of comments
desc_count -> int count of descendants
attribsl -> int number of attributes
attribs -> dict dictionary of attributes
These return None only if called from empty type. They also have _raw counterparts that return bytes e.g. text_recursive_raw -> Optional[bytes], name_raw -> Optional[bytes]
insides -> Optional[str] string containing contents inside tag or comment
name -> Optional[str] tag name e.g. 'div'
starttag -> Optional[str] head of the tag e.g. '<div class="user">'
endtag -> Optional[str] tail of the tag e.g. '</div>'
endtag_strip -> Optional[str] tail of the tag, stripped of < and > e.g. '/div'
text -> Optional[str] text of children
text_recursive -> Optional[str] text of descendants
rq = reliq("""
<main>
<ul>
<a>
<li>L1</li>
</a>
<li>L2</li>
</ul>
</main>
""")
ul = rq[0][0]
a = ul[0]
li1 = a[0]
li2 = ul[1]
ul.name
# 'ul'
ul.name_raw
# b'ul'
ul.lvl
# 1
li1.lvl
# 3
ul.text
# '\n \n \n '
ul.text_recursive
# '\n \n L1\n \n L2\n '
a.insides
# '\n <li>L1</li>\n '
Comments can either return their string representation or insides by insides property.
c = reliq('<!-- Comment -->').self(type=None)[0]
c.insides
# ' Comment '
bytes(c)
# b'<!-- Comment -->'
str(c)
# '<!-- Comment -->'
Text can only be converted to string
t = reliq('Example').self(type=None)[0]
str(t)
# 'Example'
Convert reliq objects into a list or a generator of single type objects.
If their first argument is set to True or gen=True is passed, a generator is returned, otherwise a list.
By default they filter node types to only reliq.Type.tag, this can be changed by setting the type argument e.g. type=reliq.Type.comment|reliq.Type.texterr. If type is set to None all types are matched.
If rel=True is passed returned objects will be relative to object from which they were matched.
rq = reliq("""
<!DOCTYPE html>
<head>
<title>Title</title>
</head>
<body>
<section>
<h1>Title</h1>
<p>A</p>
</section>
<h2>List</h2>
<ul>
<li>A</li>
<li>B</li>
<li>C</li>
</ul>
<section>
TEXT
</section>
</body>
""")
everything() gets all elements in structure, no matter the previous context.
#traverse everything through generator
for i in rq.everything(True):
print(str(i))
self() gets the context itself, single element for single type, list of the list type and elements with .lvl == 0 for struct type.
By default filtered type depends on object type it was called for, for single and list types are unfiltered, only struct type enforces type=reliq.Type.tag.
# rq is a reliq.Type.struct object
rq.self()
# [<tag head>, <tag body>]
rq.self(type=None)
# [<textempty>, <comment>, <textempty>, <tag head>, <textempty>, <tag body>]
rq.self(type=reliq.Type.tag|reliq.Type.comment)
# [<comment>,<tag head>, <tag body>]
# ls is a reliq.Type.list object that has comments and text types
ls = rq.filter('[:3] ( comment@ * )( text@ * )')
ls.self()
# [<comment>, <text>, <text>, <text>]
ls.self(type=reliq.Type.tag|reliq.Type.comment)
# [<comment>]
# body is a reliq.Type.single object
body = rq[1].self()
len(body.self())
# 1
body.self()[0].name
# "body"
children() gets all nodes of the context that have level relative to them equal to 1.
# struct
rq.children()
# [<tag title>, <tag section>, <tag h2>, <tag ul>, <tag section>]
# list
rq.filter('head, ul').children()
# [<tag title>, <tag li>, <tag li>, <tag li>]
# single
first_section = rq[1][0]
first_section.children()
# [<tag h1>, <tag p>]
descendants() gets all nodes of the context that have level relative to them greater or equal to 1.
# struct
rq.descendants()
# [<tag title>, <tag section>, <tag h1>, <tag p>, <tag h2>, <tag ul>, <tag li>, <tag li>, <tag li>, <tag section>]
# list
rq.filter('[0] section').descendants()
# [<tag h1>, <tag p>]
# single
rq[1][0].descendants()
# [<tag h1>, <tag p>]
full() gets all nodes of the context and all nodes below them (like calling self() and descendants() at once).
# struct
rq.full()
# [<tag head>, <tag title>, <tag body>, <tag section>, <tag h1>, <tag p>, <tag h2>, <tag ul>, <tag li>, <tag li>, <tag li>, <tag section>]
# list
rq.filter('[0] section').descendants()
# [<tag section>, <tag h1>, <tag p>]
# single
rq[1][0].descendants()
# [<tag section>, <tag h1>, <tag p>]
parent() gets parent of context nodes. Doesn't work for struct type.
# list
rq.filter('li').parent()
# [<tag ul>, <tag ul>, <tag ul>]
# single
rq[1][2][0].parent()
# [<tag li>]
# single
rq[0].parent() # top level nodes don't have parents
# []
rparent() behaves like parent() but returns the parent to which the current object is relative to. Doesn't work for struct type.
It doesn't take rel argument, returned objects are always relative.
ancestors() gets ancestors of context nodes. Doesn't work for struct type.
# list
rq.filter('li').ancestors()
# [<tag ul>, <tag body>, <tag ul>, <tag body>, <tag ul>, <tag body>]
# single
rq[1][2][0].ancestors()
# [<tag ul>, <tag body>]
# first element of ancestors() should be the same as for parent()
rq[1][2][0].ancestors()[0].name == rq[1][2][0].parent()[0].name
# single
rq[0].ancestors() # top level nodes don't have ancestors
# []
before() gets all nodes that have lower .position property than context nodes. Doesn't work for struct type.
# list
rq.filter('[0] title, [1] section').before()
# [<tag head>, <tag li>, <tag li>, <tag li>, <tag ul>, <tag h2>, <tag p>, <tag h1>, <tag section>, <tag body>, <tag title>, <tag head>]
# single
title = rq[0][0]
title.before()
# [<tag head>]
# single
second_section = rq[1][3]
second_section.before()
# [<tag li>, <tag li>, <tag li>, <tag ul>, <tag h2>, <tag p>, <tag h1>, <tag section>, <tag body>, <tag title>, <tag head>]
# single
head = rq[0]
head.before() #first element doesn't have any nodes before it
# []
preceding() is similar to before() but ignores ancestors. Doesn't work for struct type.
# list
rq.filter('[0] title, [1] section').preceding()
# [<tag li>, <tag li>, <tag li>, <tag ul>, <tag h2>, <tag p>, <tag h1>, <tag section>, <tag title>, <tag head>]
# single
title = rq[0][0]
title.preceding() # all tags before it are it's ancestors
# []
# single
second_section = rq[1][3]
second_section.preceding()
# [<tag li>, <tag li>, <tag li>, <tag ul>, <tag h2>, <tag p>, <tag h1>, <tag section>, <tag title>, <tag head>]
after() gets all nodes that have higher .position property than context nodes. Doesn't work for struct type.
# list
rq.filter('h2, ul').after()
# [<tag ul>, <tag li>, <tag li>, <tag li>, <tag section>, <tag li>, <tag li>, <tag li>, <tag section>]
# single
h2 = rq[1][1]
h2.after()
# [<tag ul>, <tag li>, <tag li>, <tag li>, <tag section>]
# single
ul = rq[1][2]
ul.after()
# [<tag li>, <tag li>, <tag li>, <tag section>]
# single
third_section = rq[1][3] # last element
third_section.after()
# []
subsequent() is similar to after() but ignores descendants. Doesn't work for struct type.
# list
rq.filter('h2, ul').subsequent()
# [<tag ul>, <tag li>, <tag li>, <tag li>, <tag section>, <tag section>]
# single
h2 = rq[1][1]
h2.subsequent()
# [<tag ul>, <tag li>, <tag li>, <tag li>, <tag section>]
# single
ul = rq[1][2]
ul.subsequent()
# [<tag section>]
siblings_preceding() gets nodes on the same level as context nodes but before them and limited to their parent. Doesn't work for struct type.
If full=True is passed descendants of siblings will also be matched.
# list
rq.filter('ul, h2').siblings_preceding()
# [<tag h2>, <tag section>, <tag section>]
# single
h2 = rq[1][1]
h2.siblings_preceding()
# [<tag section>]
h2.siblings_preceding(full=True)
# [<tag p>, <tag h1>, <tag section>]
# single
ul = rq[1][2]
ul.siblings_preceding()
# [<tag h2>, <tag section>]
ul.siblings_preceding(full=True)
# [<tag h2>, <tag p>, <tag h1>, <tag section>]
siblings_preceding() gets nodes on the same level as context nodes but after them and limited to their parent. Doesn't work for struct type.
If full=True is passed descendants of siblings will also be matched.
# list
rq.filter('ul, h2').siblings_subsequent()
# [<tag h2>, <tag section>, <tag section>]
# single
h2 = rq[1][1]
h2.siblings_subsequent()
# [<tag ul>, <tag section>]
h2.siblings_subsequent(full=True)
# [<tag ul>, <tag li>, <tag li>, <tag li>, <tag section>]
# single
ul = rq[1][2]
ul.siblings_subsequent()
# [<tag section>]
ul.siblings_subsequent(full=True)
# [<tag section>]
siblings() returns merged output of siblings_preceding() and siblings_subsequent().
reliq.expr is a class that compiles expressions, it accepts only one argument that can be a str(), bytes() or Path().
If Path() argument is specified, file under it will be read with Path.read_bytes().
# str
reliq.expr(r'table; { tr .name; li | "%(title)v\n", th }')
# bytes
reliq.expr(rb'li')
# file from Path
reliq.expr(Path('expression.reliq'))
search() executes expression in the first argument and returns str() or bytes if second argument is True or raw=True.
Expression can be passed both as compiled object of reliq.expr or its representation in str(), bytes() or Path() that will be compiled in function.
rq = reliq('<span class=name data-user-id=1282>User</span><p>Title: N1ase</p>')
rq.search(r'p')
# '<p>Title: N1ase</p>\n'
rq.search(r'p', True)
# b'<p>Title: N1ase</p>\n'
rq.search(r'p', raw=True)
# b'<p>Title: N1ase</p>\n'
rq.search(r"""
span .name; {
.id.u @ | "%(data-user-id)v",
.name @ | "%t"
},
.title p | "%i" sed "s/^Title: //"
""",True)
# b'{"id":1282,"name":"User","title":"N1ase"}'
rq.search(Path('expression.reliq'))
Same as search() but returns dict().
filter() executes expression in the first argument and returns reliq object of list type or empty type if nothing has been found.
If second argument is True or independent=True then returned object will be completely independent from the one the function was called on. A new HTML string representation will be created, and structure will be copied and shifted to new string, levels will also change.
Expression can be passed both as compiled object of reliq.expr or its representation in str(), bytes() or Path() that will be compiled in function.
Any field, formatting or string conversion in expression will be ignored, only objects used in them will be returned.
rq = reliq('<span class=name data-user-id=1282>User</span><p>Title: N1ase</p>')
rq.filter(r'p').self()
# [<tag p>]
rq.filter(r'p').type
# reliq.Type.list
rq.filter(r'p').get_data()
# '<span class=name data-user-id=1282>User</span><p>Title: N1ase</p>'
rq.filter(r'p',True).get_data()
# '<p>Title: N1ase</p>'
rq.filter(r'nothing').type
# reliq.Type.empty
rq.filter(r"""
span .name; {
.id.u @ | "%(data-user-id)v",
.name @ | "%t"
},
.title p | "%i" sed "s/^Title: //"
""")
# [<tag span>, <tag span>, <tag p>]
rq.filter(Path('expression.reliq'))
decode() decodes html entities in first argument of str() or bytes(), and returns str() or bytes() if second argument is True or raw=True.
By default is translated to space, this can be changed by setting no_nbsp=False.
encode() does the opposite of decode() in the same fashion.
By default only special characters are encoded i.e. <, >, ", ', &. If full=True is set everything possible will be converted to html entities (quite slow approach).
reliq.decode(r"text & < ⃛ Ô")
# "loop & < ⃛⃛ Ô"
reliq.decode(r"text & < ⃛ Ô",True)
# b'text & < \xe2\x83\x9b\xe2\x83\x9b \xc3\x94'
reliq.decode(r"text & < ⃛ Ô",raw=True)
# b'text & < \xe2\x83\x9b\xe2\x83\x9b \xc3\x94'
reliq.decode('ex t')
# "ex t"
reliq.decode('ex t',no_nbsp=False)
# 'ex\xa0t'
reliq.decode('ex t',True,no_nbsp=False)
# b'ex\xc2\xa0t'
reliq.encode("<p>li & \t 'seq' \n </p>")
# '<p>li &amp; \t 'seq' \n </p>'
reliq.encode("<p>li & \t 'seq' \n </p>",True)
# b'<p>li &amp; \t 'seq' \n </p>'
reliq.encode("<p>li & \t 'seq' \n </p>",raw=True)
# b'<p>li &amp; \t 'seq' \n </p>'
reliq.encode("<p>li & \t 'seq' \n </p>",full=True)
# '<p>li &amp; 	 'seq' 
 </p>'
reliq.encode("<p>li & \t 'seq' \n </p>",True,full=True)
# b'<p>li &amp; 	 'seq' 
 </p>'
All errors are instances of reliq.Error.
reliq.SystemError is raised when kernel fails (you should assume it doesn't happen).
reliq.HtmlError is raised when html structure exceeds limits.
reliq.ScriptError is raised when incorrect script is compiled.
try:
reliq('<div>'*8193)
except reliq.HtmlError:
print('html depth limit exceeded')
try:
reliq.expr('| |')
except reliq.ScriptError:
print('incorrect expression')
list and single type object also stores a pointer to node that object is relative to in context i.e. rq.filter(r'body; nav') will return nav objects that were found in body tags, nav objects might not be direct siblings of body tags but because of relativity their relation is not lost.
reliq.filter() always keeps the relativity.
By default axis functions don't change relativity unless rel=True is passed.
rq = reliq("""
<body>
<nav>
<ul>
<li> A </li>
<li> B </li>
<li> C </li>
</ul>
</nav>
</body>
""")
li = rq[0][0][0][1] # not relative
li_self = rq.filter('li i@w>"B"')[0] # relative to itself
li_rel = rq.filter('nav; li i@w>"B"')[0] # relative to nav
# .rlvl and .rposition for non relative objects return same values as .lvl and .position
li.lvl
# 3
li_rel.lvl
# 3
li.rlvl
# 3
li_rel.rlvl
# 2
li.position
# 10
li_rel.position
# 10
li.rposition
# 10
li_rel.rposition
# 9
nav = rq[0][0]
for i in nav.descendants(rel=True):
if i.rlvl == 2 and i.name == 'li':
print(i.lvl,i.rlvl)
# 3 2
break
nav_rel = li_rel.rparent()[0] # nav element relative to li
nav_rel.rlvl
# -2
nav_rel.rposition
# -7
FAQs
Python ctypes bindings for reliq
We found that reliq demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Malicious Koishi plugin silently exfiltrates messages with hex strings to a hardcoded QQ account, exposing secrets in chatbots across platforms.
Research
Security News
Malicious PyPI checkers validate stolen emails against TikTok and Instagram APIs, enabling targeted account attacks and dark web credential sales.
Security News
The Node.js TSC opted not to endorse a feature bounty program, citing concerns about incentives, governance, and project neutrality.