mgdm/htmlq: Like jq, but for HTML.


Like jq, but for HTML. Uses CSS selectors to extract bits content from HTML files. Mozilla’s MDN has a good reference for CSS selector syntax.


$ htmlq -h
htmlq 0.0.1
Runs CSS selectors on HTML

    htmlq [FLAGS] [OPTIONS] <selector>...

    -h, --help                 Prints help information
    -w, --ignore-whitespace    When printing text nodes, ignore those that consist entirely of whitespace
    -p, --pretty               Pretty-print the serialised output
    -t, --text                 Output only the contents of text nodes inside selected elements
    -V, --version              Prints version information

    -a, --attribute <attribute>    Only return this attribute (if present) from selected elements
    -f, --filename <FILE>          The input file. Defaults to stdin
    -o, --output <FILE>            The output file. Defaults to stdout

    <selector>...    The CSS expression to select


Using with cURL to find part of a page by ID

$ curl -s | htmlq '#get-help'
<div class="four columns mt3 mt0-l" id="get-help">
        <h4>Get help!</h4>
          <li><a href="">Documentation</a></li>
          <li><a href="">Ask a Question on the Users Forum</a></li>
          <li><a href="">Check Website Status</a></li>
        <div class="languages">
            <label class="hidden" for="language-footer">Language</label>
            <select id="language-footer">
                <option title="English (US)" value="en-US">English (en-US)</option>
<option title="French" value="fr">Français (fr)</option>
<option title="German" value="de">Deutsch (de)</option>


Find all the links in a page

$ curl -s | htmlq -a href a

Get the text content of a post

$ curl -s | htmlq  -t .main

          About NixOS

NixOS is a GNU/Linux distribution that aims to
improve the state of the art in system configuration management.  In
existing distributions, actions such as upgrades are dangerous:
upgrading a package can cause other packages to break, upgrading an
entire system is much less reliable than reinstalling from scratch,
you can’t safely test what the results of a configuration change will
be, you cannot easily undo changes to the system, and so on.  We want
to change that.  NixOS has many innovative features:


Pretty print HTML

(This is a bit of a work in progress)

$ curl -s | htmlq -p '#posts'
<section id="posts">
  <h2>I write about...
  <ul class="post-list">
      <time datetime="2019-04-29 00:%i:1556496000" pubdate="">
        29/04/2019</time><a href="">
        <h3>Debugging network connections on macOS with nettop
      <p>Using nettop to find out what network connections a program is trying to make.