<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>doc_internal/fish-riir-plan.md - fork/fish-shell @ master</title>
<meta name="description" content="doc_internal/fish-riir-plan.md in fork/fish-shell at master.">
<meta property="og:title" content="doc_internal/fish-riir-plan.md - fork/fish-shell @ master">
<meta property="og:description" content="doc_internal/fish-riir-plan.md in fork/fish-shell at master.">
<meta property="og:type" content="website">
<meta property="og:site_name" content="git.julia.blue">
<link rel="stylesheet" href="/css/style.css?v=62">
<script type="module" src="/js/main.js?v=62"></script>
</head>
<body>
<main data-ref="master" data-tip="605c466379a074cf1b5cdb2b62bc8071aab8b47b">
<noscript>
  <div class="callout">
    <p>
      hey! this is a static git repo viewer. commit diffs, file history,
      older commits and other revisions won't load without javascript.
    </p>
    <p>
      you are still able to view source files on branch/tag tips,
      but the experience is degraded.
    </p>
  </div>
</noscript>
<header class="repo"><div>
<h1>fork/fish-shell</h1>
</div>
<pre class="clone">git clone https://git.julia.blue/fork/fish-shell</pre>
</header>
<p class="snapshot-commit"><span class="commit-author">Toyosatomimi no Miko</span><span class="commit-message" data-commit="bf38e1beca48401bf9d9521e27cb3780e739f9b3">Rename FLOG, FLOGF, to lowercase flog, flogf</span><span class="snapshot-sha sha" data-commit="bf38e1beca48401bf9d9521e27cb3780e739f9b3">bf38e1bec</span><time datetime="2025-12-15T18:29:07-05:00" title="2025-12-15 18:29:07 -0500">2025-12-15</time></p>
<header class="topbar"><details class="switcher"><summary>master</summary><div class="menu"><strong>branches</strong><a class="current" href="/fork/fish-shell/ref/master/">master</a><a class="all" href="/fork/fish-shell/refs">all refs →</a></div></details><nav class="crumbs"><a href="/fork/fish-shell/">fish-shell</a> / <a href="/fork/fish-shell/ref/master/tree/doc_internal/">doc_internal</a> / <span class="cur">fish-riir-plan.md</span></nav>
<span class="view-stats"><span>5.4 KiB</span><span>79 lines</span></span><span class="actions"><a class="raw" href="/fork/fish-shell/raw/0332b2429f952011b5cbb2a253a82fcfe0fae766/doc_internal/fish-riir-plan.md">raw</a></span></header>
<div class="markdown-view"><input type="radio" name="markdown-view" id="markdown-rendered" checked><label for="markdown-rendered">rendered</label><input type="radio" name="markdown-view" id="markdown-code"><label for="markdown-code">code</label><section class="readme rendered"><p>These is a proposed port of fish-shell from C++ to Rust, and from CMake to cargo or related. This document is high level - see the <a href="/fork/fish-shell/ref/master/blob/doc_internal/rust-devel.md">Development Guide</a> for more details.</p>
<h2>Why Port</h2>
<ul>
<li>Gain access to more contributors and enable easier contributions. C++ is becoming a legacy language.</li>
<li>Free us from the annoyances of C++/CMake, and old toolchains.</li>
<li>Ensure fish continues to be perceived as modern and relevant.</li>
<li>Unlock concurrent mode (see below).</li>
</ul>
<h2>Why Rust</h2>
<ul>
<li>Rust is a systems programming language with broad platform support, a large community, and a relatively high probability of still being relevant in a decade.</li>
<li>Rust has a unique strength in its thread safety features, which is the missing piece to enable concurrent mode - see below.</li>
<li>Other languages considered:
<ul>
<li>Java, Python and the scripting family are ruled out for startup latency and memory usage reasons.</li>
<li>Go would be an awkward fit. fork is <a href="https://stackoverflow.com/questions/28370646/how-do-i-fork-a-go-process/28371586#28371586">quite the problem</a> in Go.</li>
<li>Other system languages (D, Nim, Zig...) are too niche: fewer contributors, higher risk of the language becoming irrelevant.</li>
</ul>
</li>
</ul>
<h2>Risks</h2>
<ul>
<li>Large amount of work with possible introduction of new bugs.</li>
<li>Long period of complicated builds.</li>
<li>Existing contributors will have to learn Rust.</li>
<li>As of yet unknown compatibility story for Tier 2+ platforms (Cygwin, etc).</li>
</ul>
<h2>Approach</h2>
<p>We will do an <strong>incremental port</strong> in the span of one release. We will have a period of using both C++ and Rust, and both cargo and CMake, leveraging FFI tools (see below).</p>
<p>The work will <strong>proceed on master</strong>: no long-lived branches. Tests and CI continue to pass at every commit for recent Linux and Mac. Centos7, *BSD, etc may be temporarily disabled if they prove problematic.</p>
<p>The Rust code will initially resemble the replaced C++. Fidelity to existing code is more important than Rust idiomaticity, to aid review and bisecting. But don't take this to extremes - use judgement.</p>
<p>The port will proceed "outside in." We'll start with leaf components (e.g. builtins) and proceed towards the core. Some components will have both a Rust and C++ implementation (e.g. flog), in other cases we'll change the existing C++ to invoke the new Rust implementations (builtins).</p>
<p>After porting the C++, we'll replace CMake.</p>
<p>We will continue to use wide chars, locales, gettext, printf format strings, and PCRE2. We will not change the fish scripting language at all. We will <em>not</em> use this as an opportunity to fix existing design flaws, with a few carefully chosen exceptions. See <a href="#strings">Strings</a>.</p>
<p>We will not use tokio, serde, async, or other fancy Rust frameworks initially.</p>
<h3>FFI</h3>
<p>Rust/C++ interop will use <a href="https://github.com/google/autocxx">autocxx</a>, <a href="https://cxx.rs">Cxx</a>, and possibly <a href="https://rust-lang.github.io/rust-bindgen/">bindgen</a>. I've forked these for fish (see the <a href="/fork/fish-shell/ref/master/blob/doc_internal/rust-devel.md">Development Guide</a>). Once the port is done, we will stop using them, except perhaps bindgen for PCRE2.</p>
<p>We will use <a href="https://github.com/corrosion-rs/corrosion">corrosion</a> for CMake integration.</p>
<p>Inefficiencies (e.g. extra string copying) at the FFI layer are fine, since it will all get thrown away.</p>
<p>Tests can stay in fish_tests.cpp or be moved into Rust .rs files; either is fine.</p>
<h3>Strings</h3>
<p>Rust's <code>String</code> / <code>&amp;str</code> types cannot represent non-UTF8 filenames or data using the default encoding scheme. That's why all string conversions must go through fish's encoding scheme (using the private-use area to encode invalid sequences). For example, fish cannot use <code>File::open</code> with a <code>&amp;str</code> because the decoding will be incorrect.</p>
<p>So instead of <code>String</code>, fish will use its own string type, and manage encoding and decoding as it does today. However we will make some specific changes:</p>
<ol>
<li>Drop the nul-terminated requirement. When passing <code>const wchar_t*</code> back to C++, we will allocate and copy into a nul-terminated buffer.</li>
<li>Drop support for 16-bit wchar. fish will use UTF32 on all platforms, and manage conversions itself.</li>
</ol>
<p>After the port we can consider moving to UTF-8, for memory usage reasons.</p>
<p>See the <a href="/fork/fish-shell/ref/master/blob/doc_internal/rust-devel.md">Rust Development Guide</a> for more on strings.</p>
<h3>Thread Safety</h3>
<p>Allowing <a href="https://github.com/fish-shell/fish-shell/issues/238">background functions</a> and concurrent functions has been a goal for many years. I have been nursing <a href="https://github.com/ridiculousfish/fish-shell/tree/concurrent_even_simpler">a long-lived branch</a> which allows full threaded execution. But though the changes are small, I have been reluctant to propose them, because they will make reasoning about the shell internals too complex: it is difficult in C++ to check and enforce what crosses thread boundaries.</p>
<p>This is Rust's bread and butter: we will encode thread requirements into our types, making it explicit and compiler-checked, via Send and Sync. Rust will allow turning on concurrent mode in a safe way, with a manageable increase in complexity, finally enabling this feature.</p>
<h2>Timeline</h2>
<p>Handwaving, 6 months? Frankly unknown - there's 102 remaining .cpp files of various lengths. It'll go faster as we get better at it. Peter (ridiculous_fish) is motivated to work on this, other current contributors have some Rust as well, and we may also get new contributors from the Rust community. Part of the point is to make contribution easier.</p>
<h2>Links</h2>
<ul>
<li><a href="https://wiki.archlinux.org/title/Rust_package_guidelines">Packaging Rust projects</a> from Arch Linux</li>
</ul>
</section><div class="code-panel"><pre class="code src language-markdown"><span class="code-line"><a class="ln" id="L1" href="#L1">1</a><span class="code-text">These is a proposed port of fish-shell from C++ to Rust, and from CMake to cargo or related. This document is high level - see the [Development Guide] for more details.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L2" href="#L2">2</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L3" href="#L3">3</a><span class="code-text"><a-p>##</a-p> <a-tt>Why Port</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L4" href="#L4">4</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L5" href="#L5">5</a><span class="code-text"><a-p>- </a-p>Gain access to more contributors and enable easier contributions. C++ is becoming a legacy language.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L6" href="#L6">6</a><span class="code-text"><a-p>- </a-p>Free us from the annoyances of C++/CMake, and old toolchains.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L7" href="#L7">7</a><span class="code-text"><a-p>- </a-p>Ensure fish continues to be perceived as modern and relevant.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L8" href="#L8">8</a><span class="code-text"><a-p>- </a-p>Unlock concurrent mode (see below).</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L9" href="#L9">9</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L10" href="#L10">10</a><span class="code-text"><a-p>## </a-p><a-tt>Why Rust</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L11" href="#L11">11</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L12" href="#L12">12</a><span class="code-text"><a-p>- Rust is a systems programming language with broad platform support, a large community, and a relatively high probability of still being relevant in a decade.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L13" href="#L13">13</a><span class="code-text"><a-p>- Rust has a unique strength in its thread safety features, which is the missing piece to enable concurrent mode - see below.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L14" href="#L14">14</a><span class="code-text"><a-p>- Other languages considered:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L15" href="#L15">15</a><span class="code-text"><a-p>  - Java, Python and the scripting family are ruled out for startup latency and memory usage reasons.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L16" href="#L16">16</a><span class="code-text"><a-p>  - Go would be an awkward fit. fork is [quite the problem](https://stackoverflow.com/questions/28370646/how-do-i-fork-a-go-process/28371586#28371586) in Go.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L17" href="#L17">17</a><span class="code-text"><a-p>  - Other system languages (D, Nim, Zig...) are too niche: fewer contributors, higher risk of the language becoming irrelevant.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L18" href="#L18">18</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L19" href="#L19">19</a><span class="code-text"><a-p>## </a-p><a-tt>Risks</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L20" href="#L20">20</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L21" href="#L21">21</a><span class="code-text"><a-p>- Large amount of work with possible introduction of new bugs.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L22" href="#L22">22</a><span class="code-text"><a-p>- Long period of complicated builds.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L23" href="#L23">23</a><span class="code-text"><a-p>- Existing contributors will have to learn Rust.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L24" href="#L24">24</a><span class="code-text"><a-p>- As of yet unknown compatibility story for Tier 2+ platforms (Cygwin, etc).</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L25" href="#L25">25</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L26" href="#L26">26</a><span class="code-text"><a-p>## </a-p><a-tt>Approach</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L27" href="#L27">27</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L28" href="#L28">28</a><span class="code-text"><a-p>We will do an **incremental port** in the span of one release. We will have a period of using both C++ and Rust, and both cargo and CMake, leveraging FFI tools (see below).</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L29" href="#L29">29</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L30" href="#L30">30</a><span class="code-text"><a-p>The work will **proceed on master**: no long-lived branches. Tests and CI continue to pass at every commit for recent Linux and Mac. Centos7, \*BSD, etc may be temporarily disabled if they prove problematic.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L31" href="#L31">31</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L32" href="#L32">32</a><span class="code-text"><a-p>The Rust code will initially resemble the replaced C++. Fidelity to existing code is more important than Rust idiomaticity, to aid review and bisecting. But don&#39;t take this to extremes - use judgement.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L33" href="#L33">33</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L34" href="#L34">34</a><span class="code-text"><a-p>The port will proceed &quot;outside in.&quot; We&#39;ll start with leaf components (e.g. builtins) and proceed towards the core. Some components will have both a Rust and C++ implementation (e.g. flog), in other cases we&#39;ll change the existing C++ to invoke the new Rust implementations (builtins).</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L35" href="#L35">35</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L36" href="#L36">36</a><span class="code-text"><a-p>After porting the C++, we&#39;ll replace CMake.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L37" href="#L37">37</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L38" href="#L38">38</a><span class="code-text"><a-p>We will continue to use wide chars, locales, gettext, printf format strings, and PCRE2. We will not change the fish scripting language at all. We will _not_ use this as an opportunity to fix existing design flaws, with a few carefully chosen exceptions. See [Strings](#strings).</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L39" href="#L39">39</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L40" href="#L40">40</a><span class="code-text"><a-p>We will not use tokio, serde, async, or other fancy Rust frameworks initially.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L41" href="#L41">41</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L42" href="#L42">42</a><span class="code-text"><a-p>### </a-p><a-tt>FFI</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L43" href="#L43">43</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L44" href="#L44">44</a><span class="code-text"><a-p>Rust/C++ interop will use [autocxx](https://github.com/google/autocxx), [Cxx](https://cxx.rs), and possibly [bindgen](https://rust-lang.github.io/rust-bindgen/). I&#39;ve forked these for fish (see the [Development Guide]). Once the port is done, we will stop using them, except perhaps bindgen for PCRE2.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L45" href="#L45">45</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L46" href="#L46">46</a><span class="code-text"><a-p>We will use [corrosion](https://github.com/corrosion-rs/corrosion) for CMake integration.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L47" href="#L47">47</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L48" href="#L48">48</a><span class="code-text"><a-p>Inefficiencies (e.g. extra string copying) at the FFI layer are fine, since it will all get thrown away.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L49" href="#L49">49</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L50" href="#L50">50</a><span class="code-text"><a-p>Tests can stay in fish_tests.cpp or be moved into Rust .rs files; either is fine.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L51" href="#L51">51</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L52" href="#L52">52</a><span class="code-text"><a-p>### </a-p><a-tt>Strings</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L53" href="#L53">53</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L54" href="#L54">54</a><span class="code-text"><a-p>Rust&#39;s `String` / `&amp;str` types cannot represent non-UTF8 filenames or data using the default encoding scheme. That&#39;s why all string conversions must go through fish&#39;s encoding scheme (using the private-use area to encode invalid sequences). For example, fish cannot use `File::open` with a `&amp;str` because the decoding will be incorrect.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L55" href="#L55">55</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L56" href="#L56">56</a><span class="code-text"><a-p>So instead of `String`, fish will use its own string type, and manage encoding and decoding as it does today. However we will make some specific changes:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L57" href="#L57">57</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L58" href="#L58">58</a><span class="code-text"><a-p>1. Drop the nul-terminated requirement. When passing `const wchar_t*` back to C++, we will allocate and copy into a nul-terminated buffer.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L59" href="#L59">59</a><span class="code-text"><a-p>2. Drop support for 16-bit wchar. fish will use UTF32 on all platforms, and manage conversions itself.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L60" href="#L60">60</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L61" href="#L61">61</a><span class="code-text"><a-p>After the port we can consider moving to UTF-8, for memory usage reasons.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L62" href="#L62">62</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L63" href="#L63">63</a><span class="code-text"><a-p>See the [Rust Development Guide][Development Guide] for more on strings.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L64" href="#L64">64</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L65" href="#L65">65</a><span class="code-text"><a-p>### </a-p><a-tt>Thread Safety</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L66" href="#L66">66</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L67" href="#L67">67</a><span class="code-text"><a-p>Allowing [background functions](https://github.com/fish-shell/fish-shell/issues/238) and concurrent functions has been a goal for many years. I have been nursing [a long-lived branch](https://github.com/ridiculousfish/fish-shell/tree/concurrent_even_simpler) which allows full threaded execution. But though the changes are small, I have been reluctant to propose them, because they will make reasoning about the shell internals too complex: it is difficult in C++ to check and enforce what crosses thread boundaries.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L68" href="#L68">68</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L69" href="#L69">69</a><span class="code-text"><a-p>This is Rust&#39;s bread and butter: we will encode thread requirements into our types, making it explicit and compiler-checked, via Send and Sync. Rust will allow turning on concurrent mode in a safe way, with a manageable increase in complexity, finally enabling this feature.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L70" href="#L70">70</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L71" href="#L71">71</a><span class="code-text"><a-p>## </a-p><a-tt>Timeline</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L72" href="#L72">72</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L73" href="#L73">73</a><span class="code-text"><a-p>Handwaving, 6 months? Frankly unknown - there&#39;s 102 remaining .cpp files of various lengths. It&#39;ll go faster as we get better at it. Peter (ridiculous_fish) is motivated to work on this, other current contributors have some Rust as well, and we may also get new contributors from the Rust community. Part of the point is to make contribution easier.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L74" href="#L74">74</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L75" href="#L75">75</a><span class="code-text"><a-p>## </a-p><a-tt>Links</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L76" href="#L76">76</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L77" href="#L77">77</a><span class="code-text"><a-p>- [Packaging Rust projects](https://wiki.archlinux.org/title/Rust_package_guidelines) from Arch Linux</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L78" href="#L78">78</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L79" href="#L79">79</a><span class="code-text"><a-tu>[Development Guide]</a-tu><a-p>: </a-p><a-tu>rust-devel.md</a-tu></span></span></pre></div></div>
</main>
</body>
</html>
