README
This README is the one indexed from the repository at its latest ref, not from the tag for this version.
Lexbor
Fast HTML5 Parser with CSS selectors (based on lexborisov's HTML5 parser lexbor). This is successor of myhtml and expected to be faster and use less memory. Usage is almost equal to myhtml.
Add this to your application's shard.yml:
dependencies:
lexbor:
github: kostya/lexborUsage example
require "lexbor"
html = <<-HTML
<html>
<body>
<div id="t1" class="red">
<a href="/#">O_o</a>
</div>
<div id="t2"></div>
</body>
</html>
HTML
lexbor = Lexbor.new(html)
lexbor.nodes(:div) do |node|
id = node["id"]?
if first_link = node.nodes(:a).first?
href = first_link["href"]?
link_text = first_link.inner_text
puts "div with id #{id} have link [#{link_text}](#{href})"
else
puts "div with id #{id} have no links"
end
end
# Output:
# div with id t1 have link [O_o](/#)
# div with id t2 have no links
Css selectors example
require "lexbor"
html = <<-HTML
<html>
<body>
<table id="t1">
<tr><td>Hello</td></tr>
</table>
<table id="t2">
<tr><td>123</td><td>other</td></tr>
<tr><td>foo</td><td>columns</td></tr>
<tr><td>bar</td><td>are</td></tr>
<tr><td>xyz</td><td>ignored</td></tr>
</table>
</body>
</html>
HTML
lexbor = Lexbor.new(html)
p lexbor.css("#t2 tr td:first-child").map(&.inner_text).to_a
# => ["123", "foo", "bar", "xyz"]
lexbor.css("#t2 tr td:first-child") do |node|
puts node.to_html
end
# => <td>123</td>
# => <td>foo</td>
# => <td>bar</td>
# => <td>xyz</td>
lexbor.css("#t2 tr") do |node|
node.css("td:first-child") do |node2|
puts node2.to_html
end
end
# => <td>123</td>
# => <td>foo</td>
# => <td>bar</td>
# => <td>xyz</td>
More Examples
Development Setup:
git clone https://github.com/kostya/lexbor.git
cd lexbor
crystal src/ext/build_ext.cr
crystal specBenchmark
Parse google results page(1.5Mb) 1000 times, and 5000 times css select.
lexbor-program myhtml-program crystagiri-program nokogiri-program
Running on Ryzen 3800x.
| Lang | Lib | Parse time, s | Css time, s | Memory, MiB |
|---|---|---|---|---|
| Ruby 2.7 | Nokolexbor(lexbor) | 4.07 | 1.80 | 112.5 |
| Crystal | lexbor | 4.75 | 0.80 | 12.3 |
| Crystal | myhtml(+modest) | 5.98 | 1.22 | 12.2 |
| Crystal | Crystagiri(libxml2) | 14.20 | - | 31.7 |
| Ruby 2.7 | Nokogiri(libxml2) | 18.67 | 92.69 | 398.4 |
Running on Apple M1.
| Lang | Lib | Parse time, s | Css time, s | Memory, MiB |
|---|---|---|---|---|
| Crystal | lexbor | 2.77 | 0.62 | 26.4 |
| Ruby 2.7 | Nokolexbor(lexbor) | 3.41 | 1.11 | 268.8 |
| Ruby 2.7 | Nokogiri(libxml2) | 18.22 | 87.21 | 232.8 |
| Crystal | Crystagiri(libxml2) | 126.25 | - | 26.8 |