<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>README.md - +/rssfetch @ main</title>
<meta name="description" content="README.md in +/rssfetch at main.">
<meta property="og:title" content="README.md - +/rssfetch @ main">
<meta property="og:description" content="README.md in +/rssfetch at main.">
<meta property="og:type" content="website">
<meta property="og:site_name" content="git.julia.blue">
<link rel="stylesheet" href="/css/style.css?v=62">
<script type="module" src="/js/main.js?v=62"></script>
</head>
<body>
<main data-ref="main" data-tip="964ea1a449db082cb5ef2fce9a1cd708163eff87">
<noscript>
  <div class="callout">
    <p>
      hey! this is a static git repo viewer. commit diffs, file history,
      older commits and other revisions won't load without javascript.
    </p>
    <p>
      you are still able to view source files on branch/tag tips,
      but the experience is degraded.
    </p>
  </div>
</noscript>
<header class="repo"><div>
<h1>+/rssfetch</h1>
</div>
<pre class="clone">git clone https://git.julia.blue/+/rssfetch</pre>
</header>
<p class="snapshot-commit"><span class="commit-author">Julia Ryan</span><span class="commit-message" data-commit="b519185e8db40850a49b9c2de31a6d4158b374f7">Require url schemes</span><span class="snapshot-sha sha" data-commit="b519185e8db40850a49b9c2de31a6d4158b374f7">b519185</span><time datetime="2024-07-06T22:23:53-07:00" title="2024-07-06 22:23:53 -0700">2024-07-06</time></p>
<header class="topbar"><details class="switcher"><summary>main</summary><div class="menu"><strong>branches</strong><a class="current" href="/%2B/rssfetch/ref/main/">main</a><a class="all" href="/%2B/rssfetch/refs">all refs →</a></div></details><nav class="crumbs"><a href="/%2B/rssfetch/">rssfetch</a> / <span class="cur">README.md</span></nav>
<span class="view-stats"><span>1.5 KiB</span><span>29 lines</span></span><span class="actions"><a class="raw" href="/%2B/rssfetch/raw/a46b40c260c7e4ce0cdc3a0eb87da2792503145d/README.md">raw</a></span></header>
<div class="markdown-view"><input type="radio" name="markdown-view" id="markdown-rendered" checked><label for="markdown-rendered">rendered</label><input type="radio" name="markdown-view" id="markdown-code"><label for="markdown-code">code</label><section class="readme rendered"><p>Small RSS scraping program.</p>
<p>You give it a list of blogs as jsonl:</p>
<pre class="language-json">{<a-s>&quot;title&quot;</a-s>:<a-s>&quot;Jade&quot;</a-s>, <a-s>&quot;url&quot;</a-s>:<a-s>&quot;https://jade.fyi&quot;</a-s>, <a-s>&quot;feed&quot;</a-s>:<a-s>&quot;/rss.xml&quot;</a-s>}
{<a-s>&quot;title&quot;</a-s>:<a-s>&quot;Brendan Gregg&quot;</a-s>, <a-s>&quot;url&quot;</a-s>:<a-s>&quot;https://www.brendangregg.com/blog&quot;</a-s>, <a-s>&quot;feed&quot;</a-s>:<a-s>&quot;/rss.xml&quot;</a-s>}
{<a-s>&quot;title&quot;</a-s>:<a-s>&quot;Serge Zaitsev&quot;</a-s>, <a-s>&quot;url&quot;</a-s>:<a-s>&quot;https://zserge.com&quot;</a-s>, <a-s>&quot;feed&quot;</a-s>:<a-s>&quot;/rss.xml&quot;</a-s>}
...</pre>
<p>and run <code>rssfetch blogs.jsonl &gt; posts.jsonl</code> to get a list of posts like:</p>
<pre class="language-json">{<a-s>&quot;title&quot;</a-s>:<a-s>&quot;Oh no, `git send-email`&quot;</a-s>, <a-s>&quot;url&quot;</a-s>:<a-s>&quot;https://jade.fyi/blog/oh-no-git-send-email/&quot;</a-s>, <a-s>&quot;blog_title&quot;</a-s>:<a-s>&quot;Jade&quot;</a-s>, <a-s>&quot;blog_url&quot;</a-s>:<a-s>&quot;https://jade.fyi&quot;</a-s>, <a-s>&quot;date&quot;</a-s>:<a-s>&quot;2022-04-22&quot;</a-s>}
{<a-s>&quot;title&quot;</a-s>:<a-s>&quot;Debugging: using rr to deal with unruly children (processes)&quot;</a-s>, <a-s>&quot;url&quot;</a-s>:<a-s>&quot;https://jade.fyi/blog/debugging-rr-children/&quot;</a-s>, <a-s>&quot;blog_title&quot;</a-s>:<a-s>&quot;Jade&quot;</a-s>, <a-s>&quot;blog_url&quot;</a-s>:<a-s>&quot;https://jade.fyi&quot;</a-s>, <a-s>&quot;date&quot;</a-s>:<a-s>&quot;2022-02-24&quot;</a-s>}
{<a-s>&quot;title&quot;</a-s>:<a-s>&quot;On Transpilers&quot;</a-s>, <a-s>&quot;url&quot;</a-s>:<a-s>&quot;https://zserge.com/posts/transpilers/&quot;</a-s>, <a-s>&quot;blog_title&quot;</a-s>:<a-s>&quot;Serge Zaitsev&quot;</a-s>, <a-s>&quot;blog_url&quot;</a-s>:<a-s>&quot;https://zserge.com&quot;</a-s>, <a-s>&quot;date&quot;</a-s>:<a-s>&quot;2022-07-06&quot;</a-s>}
{<a-s>&quot;title&quot;</a-s>:<a-s>&quot;Learn a language by writing too many Forths&quot;</a-s>, <a-s>&quot;url&quot;</a-s>:<a-s>&quot;https://zserge.com/posts/too-many-forths/&quot;</a-s>, <a-s>&quot;blog_title&quot;</a-s>:<a-s>&quot;Serge Zaitsev&quot;</a-s>, <a-s>&quot;blog_url&quot;</a-s>:<a-s>&quot;https://zserge.com&quot;</a-s>, <a-s>&quot;date&quot;</a-s>:<a-s>&quot;2022-07-05&quot;</a-s>}
...</pre>
<p>On stderr it'll emit progress/status info:</p>
<pre>✅ Serge Zaitsev: 78 posts
❌ Brendan Gregg: ParseError(NoFeedRoot)
✅ Jade: 62 posts
...
</pre>
<p>Some feeds only show the most recent N posts. If we want to get more than that then we'd have to keep around the posts list and update it with new entries on each run, but that'd also make things more complicated and I don't particularly care about that functionality.</p>
</section><div class="code-panel"><pre class="code src language-markdown"><span class="code-line"><a class="ln" id="L1" href="#L1">1</a><span class="code-text">Small RSS scraping program.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L2" href="#L2">2</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L3" href="#L3">3</a><span class="code-text">You give it a list of blogs as jsonl:</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L4" href="#L4">4</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L5" href="#L5">5</a><span class="code-text"><a-p>```</a-p><a-tl>json</a-tl></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L6" href="#L6">6</a><span class="code-text"><a-p>{</a-p><a-s>&quot;title&quot;</a-s><a-p>:</a-p><a-s>&quot;Jade&quot;</a-s><a-p>, </a-p><a-s>&quot;url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://jade.fyi&quot;</a-s><a-p>, </a-p><a-s>&quot;feed&quot;</a-s><a-p>:</a-p><a-s>&quot;/rss.xml&quot;</a-s><a-p>}</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L7" href="#L7">7</a><span class="code-text"><a-p>{</a-p><a-s>&quot;title&quot;</a-s><a-p>:</a-p><a-s>&quot;Brendan Gregg&quot;</a-s><a-p>, </a-p><a-s>&quot;url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://www.brendangregg.com/blog&quot;</a-s><a-p>, </a-p><a-s>&quot;feed&quot;</a-s><a-p>:</a-p><a-s>&quot;/rss.xml&quot;</a-s><a-p>}</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L8" href="#L8">8</a><span class="code-text"><a-p>{</a-p><a-s>&quot;title&quot;</a-s><a-p>:</a-p><a-s>&quot;Serge Zaitsev&quot;</a-s><a-p>, </a-p><a-s>&quot;url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://zserge.com&quot;</a-s><a-p>, </a-p><a-s>&quot;feed&quot;</a-s><a-p>:</a-p><a-s>&quot;/rss.xml&quot;</a-s><a-p>}</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L9" href="#L9">9</a><span class="code-text"><a-p>...</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L10" href="#L10">10</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L11" href="#L11">11</a><span class="code-text"><a-p>and run `rssfetch blogs.jsonl &gt; posts.jsonl` to get a list of posts like:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L12" href="#L12">12</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L13" href="#L13">13</a><span class="code-text"><a-p>```</a-p><a-tl> json</a-tl></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L14" href="#L14">14</a><span class="code-text"><a-p>{</a-p><a-s>&quot;title&quot;</a-s><a-p>:</a-p><a-s>&quot;Oh no, `git send-email`&quot;</a-s><a-p>, </a-p><a-s>&quot;url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://jade.fyi/blog/oh-no-git-send-email/&quot;</a-s><a-p>, </a-p><a-s>&quot;blog_title&quot;</a-s><a-p>:</a-p><a-s>&quot;Jade&quot;</a-s><a-p>, </a-p><a-s>&quot;blog_url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://jade.fyi&quot;</a-s><a-p>, </a-p><a-s>&quot;date&quot;</a-s><a-p>:</a-p><a-s>&quot;2022-04-22&quot;</a-s><a-p>}</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L15" href="#L15">15</a><span class="code-text"><a-p>{</a-p><a-s>&quot;title&quot;</a-s><a-p>:</a-p><a-s>&quot;Debugging: using rr to deal with unruly children (processes)&quot;</a-s><a-p>, </a-p><a-s>&quot;url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://jade.fyi/blog/debugging-rr-children/&quot;</a-s><a-p>, </a-p><a-s>&quot;blog_title&quot;</a-s><a-p>:</a-p><a-s>&quot;Jade&quot;</a-s><a-p>, </a-p><a-s>&quot;blog_url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://jade.fyi&quot;</a-s><a-p>, </a-p><a-s>&quot;date&quot;</a-s><a-p>:</a-p><a-s>&quot;2022-02-24&quot;</a-s><a-p>}</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L16" href="#L16">16</a><span class="code-text"><a-p>{</a-p><a-s>&quot;title&quot;</a-s><a-p>:</a-p><a-s>&quot;On Transpilers&quot;</a-s><a-p>, </a-p><a-s>&quot;url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://zserge.com/posts/transpilers/&quot;</a-s><a-p>, </a-p><a-s>&quot;blog_title&quot;</a-s><a-p>:</a-p><a-s>&quot;Serge Zaitsev&quot;</a-s><a-p>, </a-p><a-s>&quot;blog_url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://zserge.com&quot;</a-s><a-p>, </a-p><a-s>&quot;date&quot;</a-s><a-p>:</a-p><a-s>&quot;2022-07-06&quot;</a-s><a-p>}</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L17" href="#L17">17</a><span class="code-text"><a-p>{</a-p><a-s>&quot;title&quot;</a-s><a-p>:</a-p><a-s>&quot;Learn a language by writing too many Forths&quot;</a-s><a-p>, </a-p><a-s>&quot;url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://zserge.com/posts/too-many-forths/&quot;</a-s><a-p>, </a-p><a-s>&quot;blog_title&quot;</a-s><a-p>:</a-p><a-s>&quot;Serge Zaitsev&quot;</a-s><a-p>, </a-p><a-s>&quot;blog_url&quot;</a-s><a-p>:</a-p><a-s>&quot;https://zserge.com&quot;</a-s><a-p>, </a-p><a-s>&quot;date&quot;</a-s><a-p>:</a-p><a-s>&quot;2022-07-05&quot;</a-s><a-p>}</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L18" href="#L18">18</a><span class="code-text"><a-p>...</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L19" href="#L19">19</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L20" href="#L20">20</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L21" href="#L21">21</a><span class="code-text"><a-p>On stderr it&#39;ll emit progress/status info:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L22" href="#L22">22</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L23" href="#L23">23</a><span class="code-text"><a-p>✅ Serge Zaitsev: 78 posts</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L24" href="#L24">24</a><span class="code-text"><a-p>❌ Brendan Gregg: ParseError(NoFeedRoot)</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L25" href="#L25">25</a><span class="code-text"><a-p>✅ Jade: 62 posts</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L26" href="#L26">26</a><span class="code-text"><a-p>...</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L27" href="#L27">27</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L28" href="#L28">28</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L29" href="#L29">29</a><span class="code-text">Some feeds only show the most recent N posts. If we want to get more than that then we&#39;d have to keep around the posts list and update it with new entries on each run, but that&#39;d also make things more complicated and I don&#39;t particularly care about that functionality.</span></span></pre></div></div>
</main>
</body>
</html>
