<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>benches/README.md - fork/cargo @ 0.88.0</title>
<meta name="description" content="benches/README.md in fork/cargo at 0.88.0.">
<meta property="og:title" content="benches/README.md - fork/cargo @ 0.88.0">
<meta property="og:description" content="benches/README.md in fork/cargo at 0.88.0.">
<meta property="og:type" content="website">
<meta property="og:site_name" content="git.julia.blue">
<link rel="stylesheet" href="/css/style.css?v=62">
<script type="module" src="/js/main.js?v=62"></script>
</head>
<body>
<main data-ref="0.88.0" data-tip="99624be96e9d213b0e9b1e36451271f24e4a41d8">
<noscript>
  <div class="callout">
    <p>
      hey! this is a static git repo viewer. commit diffs, file history,
      older commits and other revisions won't load without javascript.
    </p>
    <p>
      you are still able to view source files on branch/tag tips,
      but the experience is degraded.
    </p>
  </div>
</noscript>
<header class="repo"><div>
<h1>fork/cargo</h1>
</div>
<pre class="clone">git clone https://git.julia.blue/fork/cargo</pre>
</header>
<p class="snapshot-commit"><span class="commit-author">bors</span><span class="commit-message" data-commit="9a1b0924c6ea3ac306b663aa73347d3b6a59e7c3">Auto merge of #12634 - ehuss:last-use, r=epage</span><span class="snapshot-sha sha" data-commit="9a1b0924c6ea3ac306b663aa73347d3b6a59e7c3">9a1b0924c</span><time datetime="2023-11-11T20:45:02+00:00" title="2023-11-11 20:45:02 +0000">2023-11-11</time></p>
<header class="topbar"><details class="switcher"><summary>0.88.0</summary><div class="menu"><strong>branches</strong><a href="/fork/cargo/ref/master/">master</a><strong>tags</strong><a href="/fork/cargo/ref/0.87.0/">0.87.0</a><a class="current" href="/fork/cargo/ref/0.88.0/">0.88.0</a><a href="/fork/cargo/ref/0.89.0/">0.89.0</a><a href="/fork/cargo/ref/0.90.0/">0.90.0</a><a href="/fork/cargo/ref/0.91.0/">0.91.0</a><a href="/fork/cargo/ref/0.92.0/">0.92.0</a><a href="/fork/cargo/ref/0.93.0/">0.93.0</a><a href="/fork/cargo/ref/home-0.5.12/">home-0.5.12</a><a class="all" href="/fork/cargo/refs">all refs →</a></div></details><nav class="crumbs"><a href="/fork/cargo/">cargo</a> / <a href="/fork/cargo/ref/0.88.0/tree/benches/">benches</a> / <span class="cur">README.md</span></nav>
<span class="view-stats"><span>6.2 KiB</span><span>157 lines</span></span><span class="actions"><a class="raw" href="/fork/cargo/raw/900bf084ca3a042ddc029ee4f34176b7297cf87d/benches/README.md">raw</a></span></header>
<div class="markdown-view"><input type="radio" name="markdown-view" id="markdown-rendered" checked><label for="markdown-rendered">rendered</label><input type="radio" name="markdown-view" id="markdown-code"><label for="markdown-code">code</label><section class="readme rendered"><h1>Cargo Benchmarking</h1>
<p>This directory contains some benchmarks for cargo itself. This uses
<a href="https://bheisler.github.io/criterion.rs/book/">Criterion</a> for running benchmarks. It is recommended to read the Criterion
book to get familiar with how to use it. A basic usage would be:</p>
<pre class="language-bash"><a-f>cd</a-f> benches/benchsuite
<a-f>cargo</a-f> bench</pre>
<p>However, running all benchmarks would take many minutes, so in most cases it
is recommended to just run the benchmarks relevant to whatever section of code
you are working on.</p>
<h2>Benchmarks</h2>
<p>There are several different kinds of benchmarks in the <code>benchsuite/benches</code> directory:</p>
<ul>
<li><code>global_cache_tracker</code> — Benchmarks saving data to the global cache tracker
database using samples of real-world data.</li>
<li><code>resolve</code> — Benchmarks the resolver against simulations of real-world workspaces.</li>
<li><code>workspace_initialization</code> — Benchmarks initialization of a workspace
against simulations of real-world workspaces.</li>
</ul>
<h3>Resolve benchmarks</h3>
<p>The resolve benchmarks involve downloading the index and benchmarking against some
real-world and artificial workspaces located in the <a href="/fork/cargo/ref/0.88.0/tree/benches/workspaces/"><code>workspaces</code></a>
directory.</p>
<p><strong>Beware</strong> that the initial download can take a fairly long amount of time (10
minutes minimum on an extremely fast network) and require significant disk
space (around 4.5GB). The benchsuite will cache the index and downloaded
crates in the <code>target/tmp/bench</code> directory, so subsequent runs should be
faster. You can (and probably should) specify individual benchmarks to run to
narrow it down to a more reasonable set, for example:</p>
<pre class="language-bash"><a-f>cargo</a-f> bench <a-co>-p</a-co> benchsuite <a-co>--bench</a-co> resolve <a-co>--</a-co> resolve_ws/rust</pre>
<p>This will only download what's necessary for the rust-lang/rust workspace
(which is about 330MB) and run the benchmarks against it (which should take
about a minute). To get a list of all the benchmarks, run:</p>
<pre class="language-bash"><a-f>cargo</a-f> bench <a-co>-p</a-co> benchsuite <a-co>--bench</a-co> resolve <a-co>--</a-co> <a-co>--list</a-co></pre>
<h3>Global cache tracker</h3>
<p>The <code>global_cache_tracker</code> benchmark tests saving data to the global cache
tracker database using samples of real-world data. This benchmark should run
relatively quickly.</p>
<p>The real-world data is based on a capture of my personal development
environment which has accumulated a large cache. So it is somewhat arbitrary,
but hopefully representative of a challenging environment. Capturing of the
data is done with the <code>capture-last-use</code> binary, which you can run if you need
to rebuild the database. Just try to run on a system with a relatively full
cache in your cargo home directory.</p>
<pre class="language-bash"><a-f>cargo</a-f> bench <a-co>-p</a-co> benchsuite <a-co>--bench</a-co> global_cache_tracker</pre>
<h2>Viewing reports</h2>
<p>The benchmarks display some basic information on the command-line while they
run. A more complete HTML report can be found at
<code>target/criterion/report/index.html</code> which contains links to all the
benchmarks and summaries. Check out the Criterion book for more information on
the extensive reporting capabilities.</p>
<h2>Comparing implementations</h2>
<p>Knowing the raw numbers can be useful, but what you're probably most
interested in is checking if your changes help or hurt performance. To do
that, you need to run the benchmarks multiple times.</p>
<p>First, run the benchmarks from the master branch of cargo without any changes.
To make it easier to compare, Criterion supports naming the baseline so that
you can iterate on your code and compare against it multiple times.</p>
<pre class="language-bash"><a-f>cargo</a-f> bench <a-co>--</a-co> <a-co>--save-baseline</a-co> master</pre>
<p>Now you can switch to your branch with your changes. Re-run the benchmarks
compared against the baseline:</p>
<pre class="language-bash"><a-f>cargo</a-f> bench <a-co>--</a-co> <a-co>--baseline</a-co> master</pre>
<p>You can repeat the last command as you make changes to re-compare against the
master baseline.</p>
<p>Without the baseline arguments, it will compare against the last run, which
can be helpful for comparing incremental changes.</p>
<h2>Capturing workspaces</h2>
<p>The <a href="/fork/cargo/ref/0.88.0/tree/benches/workspaces/"><code>workspaces</code></a> directory contains several workspaces that
provide a variety of different workspaces intended to provide good exercises
for benchmarks. Some of these are shadow copies of real-world workspaces. This
is done with the tool in the <a href="/fork/cargo/ref/0.88.0/tree/benches/capture/"><code>capture</code></a> directory. The tool will
copy <code>Cargo.lock</code> and all of the <code>Cargo.toml</code> files of the workspace members.
It also adds an empty <code>lib.rs</code> so Cargo won't error, and sanitizes the
<code>Cargo.toml</code> to some degree, removing unwanted elements. Finally, it
compresses everything into a <code>tgz</code>.</p>
<p>To run it, do:</p>
<pre class="language-bash"><a-f>cd</a-f> benches/capture
<a-f>cargo</a-f> run <a-co>--</a-co> /path/to/workspace/foo</pre>
<p>The resolver benchmarks also support the <code>CARGO_BENCH_WORKSPACES</code> environment
variable, which you can point to a Cargo workspace if you want to try
different workspaces. For example:</p>
<pre class="language-bash"><a-pr>CARGO_BENCH_WORKSPACES</a-pr>=/path/to/some/workspace <a-f>cargo</a-f> bench</pre>
<h2>TODO</h2>
<p>This is just a start for establishing a benchmarking suite for Cargo. There's
a lot that can be added. Some ideas:</p>
<ul>
<li>Fix the benchmarks so that the resolver setup doesn't run every iteration.</li>
<li>Benchmark <a href="https://github.com/rust-lang/cargo/blob/a821e2cb24d7b6013433f069ab3bad53d160e100/src/cargo/ops/cargo_compile.rs#L470-L549">this section of
code</a>
which builds the unit graph. The performance there isn't great, and it would
be good to keep an eye on it. Unfortunately that would mean doing a bit of
work to make <code>generate_targets</code> publicly visible, and there is a bunch of
setup code that may need to be duplicated.</li>
<li>Benchmark the fingerprinting code.</li>
<li>Benchmark running the <code>cargo</code> executable. Running something like <code>cargo build</code> or <code>cargo check</code> with everything "Fresh" would be a good end-to-end
exercise to measure the overall overhead of Cargo.</li>
<li>Benchmark pathological resolver scenarios. There might be some cases where
the resolver can spend a significant amount of time. It would be good to
identify if these exist, and create benchmarks for them. This may require
creating an artificial index, similar to the <code>resolver-tests</code>. This should
also consider scenarios where the resolver ultimately fails.</li>
<li>Benchmark without <code>Cargo.lock</code>. I'm not sure if this is particularly
valuable, since we are mostly concerned with incremental builds which will
always have a lock file.</li>
<li>Benchmark just
<a href="https://github.com/rust-lang/cargo/blob/a821e2cb24d7b6013433f069ab3bad53d160e100/src/cargo/core/resolver/mod.rs#L122"><code>resolve::resolve</code></a>
without anything else. This can help focus on just the resolver.</li>
</ul>
</section><div class="code-panel"><pre class="code src language-markdown"><span class="code-line"><a class="ln" id="L1" href="#L1">1</a><span class="code-text"><a-p>#</a-p> <a-tt>Cargo Benchmarking</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L2" href="#L2">2</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L3" href="#L3">3</a><span class="code-text">This directory contains some benchmarks for cargo itself. This uses</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L4" href="#L4">4</a><span class="code-text">[Criterion] for running benchmarks. It is recommended to read the Criterion</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L5" href="#L5">5</a><span class="code-text">book to get familiar with how to use it. A basic usage would be:</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L6" href="#L6">6</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L7" href="#L7">7</a><span class="code-text"><a-p>```</a-p><a-tl>sh</a-tl></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L8" href="#L8">8</a><span class="code-text"><a-p>cd benches/benchsuite</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L9" href="#L9">9</a><span class="code-text"><a-p>cargo bench</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L10" href="#L10">10</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L11" href="#L11">11</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L12" href="#L12">12</a><span class="code-text"><a-p>However, running all benchmarks would take many minutes, so in most cases it</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L13" href="#L13">13</a><span class="code-text"><a-p>is recommended to just run the benchmarks relevant to whatever section of code</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L14" href="#L14">14</a><span class="code-text"><a-p>you are working on.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L15" href="#L15">15</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L16" href="#L16">16</a><span class="code-text"><a-p>## </a-p><a-tt>Benchmarks</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L17" href="#L17">17</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L18" href="#L18">18</a><span class="code-text"><a-p>There are several different kinds of benchmarks in the `benchsuite/benches` directory:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L19" href="#L19">19</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L20" href="#L20">20</a><span class="code-text"><a-p>* `global_cache_tracker` — Benchmarks saving data to the global cache tracker</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L21" href="#L21">21</a><span class="code-text"><a-p>  database using samples of real-world data.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L22" href="#L22">22</a><span class="code-text"><a-p>* `resolve` — Benchmarks the resolver against simulations of real-world workspaces.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L23" href="#L23">23</a><span class="code-text"><a-p>* `workspace_initialization` — Benchmarks initialization of a workspace</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L24" href="#L24">24</a><span class="code-text"><a-p>  against simulations of real-world workspaces.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L25" href="#L25">25</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L26" href="#L26">26</a><span class="code-text"><a-p>### </a-p><a-tt>Resolve benchmarks</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L27" href="#L27">27</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L28" href="#L28">28</a><span class="code-text"><a-p>The resolve benchmarks involve downloading the index and benchmarking against some</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L29" href="#L29">29</a><span class="code-text"><a-p>real-world and artificial workspaces located in the [`workspaces`](workspaces)</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L30" href="#L30">30</a><span class="code-text"><a-p>directory.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L31" href="#L31">31</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L32" href="#L32">32</a><span class="code-text"><a-p>**Beware** that the initial download can take a fairly long amount of time (10</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L33" href="#L33">33</a><span class="code-text"><a-p>minutes minimum on an extremely fast network) and require significant disk</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L34" href="#L34">34</a><span class="code-text"><a-p>space (around 4.5GB). The benchsuite will cache the index and downloaded</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L35" href="#L35">35</a><span class="code-text"><a-p>crates in the `target/tmp/bench` directory, so subsequent runs should be</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L36" href="#L36">36</a><span class="code-text"><a-p>faster. You can (and probably should) specify individual benchmarks to run to</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L37" href="#L37">37</a><span class="code-text"><a-p>narrow it down to a more reasonable set, for example:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L38" href="#L38">38</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L39" href="#L39">39</a><span class="code-text"><a-p>```</a-p><a-tl>sh</a-tl></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L40" href="#L40">40</a><span class="code-text"><a-p>cargo bench -p benchsuite --bench resolve -- resolve_ws/rust</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L41" href="#L41">41</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L42" href="#L42">42</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L43" href="#L43">43</a><span class="code-text"><a-p>This will only download what&#39;s necessary for the rust-lang/rust workspace</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L44" href="#L44">44</a><span class="code-text"><a-p>(which is about 330MB) and run the benchmarks against it (which should take</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L45" href="#L45">45</a><span class="code-text"><a-p>about a minute). To get a list of all the benchmarks, run:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L46" href="#L46">46</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L47" href="#L47">47</a><span class="code-text"><a-p>```</a-p><a-tl>sh</a-tl></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L48" href="#L48">48</a><span class="code-text"><a-p>cargo bench -p benchsuite --bench resolve -- --list</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L49" href="#L49">49</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L50" href="#L50">50</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L51" href="#L51">51</a><span class="code-text"><a-p>### </a-p><a-tt>Global cache tracker</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L52" href="#L52">52</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L53" href="#L53">53</a><span class="code-text"><a-p>The `global_cache_tracker` benchmark tests saving data to the global cache</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L54" href="#L54">54</a><span class="code-text"><a-p>tracker database using samples of real-world data. This benchmark should run</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L55" href="#L55">55</a><span class="code-text"><a-p>relatively quickly.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L56" href="#L56">56</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L57" href="#L57">57</a><span class="code-text"><a-p>The real-world data is based on a capture of my personal development</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L58" href="#L58">58</a><span class="code-text"><a-p>environment which has accumulated a large cache. So it is somewhat arbitrary,</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L59" href="#L59">59</a><span class="code-text"><a-p>but hopefully representative of a challenging environment. Capturing of the</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L60" href="#L60">60</a><span class="code-text"><a-p>data is done with the `capture-last-use` binary, which you can run if you need</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L61" href="#L61">61</a><span class="code-text"><a-p>to rebuild the database. Just try to run on a system with a relatively full</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L62" href="#L62">62</a><span class="code-text"><a-p>cache in your cargo home directory.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L63" href="#L63">63</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L64" href="#L64">64</a><span class="code-text"><a-p>```</a-p><a-tl>sh</a-tl></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L65" href="#L65">65</a><span class="code-text"><a-p>cargo bench -p benchsuite --bench global_cache_tracker</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L66" href="#L66">66</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L67" href="#L67">67</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L68" href="#L68">68</a><span class="code-text"><a-p>## </a-p><a-tt>Viewing reports</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L69" href="#L69">69</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L70" href="#L70">70</a><span class="code-text"><a-p>The benchmarks display some basic information on the command-line while they</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L71" href="#L71">71</a><span class="code-text"><a-p>run. A more complete HTML report can be found at</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L72" href="#L72">72</a><span class="code-text"><a-p>`target/criterion/report/index.html` which contains links to all the</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L73" href="#L73">73</a><span class="code-text"><a-p>benchmarks and summaries. Check out the Criterion book for more information on</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L74" href="#L74">74</a><span class="code-text"><a-p>the extensive reporting capabilities.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L75" href="#L75">75</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L76" href="#L76">76</a><span class="code-text"><a-p>## </a-p><a-tt>Comparing implementations</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L77" href="#L77">77</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L78" href="#L78">78</a><span class="code-text"><a-p>Knowing the raw numbers can be useful, but what you&#39;re probably most</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L79" href="#L79">79</a><span class="code-text"><a-p>interested in is checking if your changes help or hurt performance. To do</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L80" href="#L80">80</a><span class="code-text"><a-p>that, you need to run the benchmarks multiple times.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L81" href="#L81">81</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L82" href="#L82">82</a><span class="code-text"><a-p>First, run the benchmarks from the master branch of cargo without any changes.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L83" href="#L83">83</a><span class="code-text"><a-p>To make it easier to compare, Criterion supports naming the baseline so that</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L84" href="#L84">84</a><span class="code-text"><a-p>you can iterate on your code and compare against it multiple times.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L85" href="#L85">85</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L86" href="#L86">86</a><span class="code-text"><a-p>```</a-p><a-tl>sh</a-tl></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L87" href="#L87">87</a><span class="code-text"><a-p>cargo bench -- --save-baseline master</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L88" href="#L88">88</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L89" href="#L89">89</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L90" href="#L90">90</a><span class="code-text"><a-p>Now you can switch to your branch with your changes. Re-run the benchmarks</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L91" href="#L91">91</a><span class="code-text"><a-p>compared against the baseline:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L92" href="#L92">92</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L93" href="#L93">93</a><span class="code-text"><a-p>```</a-p><a-tl>sh</a-tl></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L94" href="#L94">94</a><span class="code-text"><a-p>cargo bench -- --baseline master</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L95" href="#L95">95</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L96" href="#L96">96</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L97" href="#L97">97</a><span class="code-text"><a-p>You can repeat the last command as you make changes to re-compare against the</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L98" href="#L98">98</a><span class="code-text"><a-p>master baseline.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L99" href="#L99">99</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L100" href="#L100">100</a><span class="code-text"><a-p>Without the baseline arguments, it will compare against the last run, which</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L101" href="#L101">101</a><span class="code-text"><a-p>can be helpful for comparing incremental changes.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L102" href="#L102">102</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L103" href="#L103">103</a><span class="code-text"><a-p>## </a-p><a-tt>Capturing workspaces</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L104" href="#L104">104</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L105" href="#L105">105</a><span class="code-text"><a-p>The [`workspaces`](workspaces) directory contains several workspaces that</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L106" href="#L106">106</a><span class="code-text"><a-p>provide a variety of different workspaces intended to provide good exercises</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L107" href="#L107">107</a><span class="code-text"><a-p>for benchmarks. Some of these are shadow copies of real-world workspaces. This</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L108" href="#L108">108</a><span class="code-text"><a-p>is done with the tool in the [`capture`](capture) directory. The tool will</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L109" href="#L109">109</a><span class="code-text"><a-p>copy `Cargo.lock` and all of the `Cargo.toml` files of the workspace members.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L110" href="#L110">110</a><span class="code-text"><a-p>It also adds an empty `lib.rs` so Cargo won&#39;t error, and sanitizes the</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L111" href="#L111">111</a><span class="code-text"><a-p>`Cargo.toml` to some degree, removing unwanted elements. Finally, it</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L112" href="#L112">112</a><span class="code-text"><a-p>compresses everything into a `tgz`.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L113" href="#L113">113</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L114" href="#L114">114</a><span class="code-text"><a-p>To run it, do:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L115" href="#L115">115</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L116" href="#L116">116</a><span class="code-text"><a-p>```</a-p><a-tl>sh</a-tl></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L117" href="#L117">117</a><span class="code-text"><a-p>cd benches/capture</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L118" href="#L118">118</a><span class="code-text"><a-p>cargo run -- /path/to/workspace/foo</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L119" href="#L119">119</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L120" href="#L120">120</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L121" href="#L121">121</a><span class="code-text"><a-p>The resolver benchmarks also support the `CARGO_BENCH_WORKSPACES` environment</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L122" href="#L122">122</a><span class="code-text"><a-p>variable, which you can point to a Cargo workspace if you want to try</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L123" href="#L123">123</a><span class="code-text"><a-p>different workspaces. For example:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L124" href="#L124">124</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L125" href="#L125">125</a><span class="code-text"><a-p>```</a-p><a-tl>sh</a-tl></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L126" href="#L126">126</a><span class="code-text"><a-p>CARGO_BENCH_WORKSPACES=/path/to/some/workspace cargo bench</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L127" href="#L127">127</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L128" href="#L128">128</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L129" href="#L129">129</a><span class="code-text"><a-p>## </a-p><a-tt>TODO</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L130" href="#L130">130</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L131" href="#L131">131</a><span class="code-text"><a-p>This is just a start for establishing a benchmarking suite for Cargo. There&#39;s</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L132" href="#L132">132</a><span class="code-text"><a-p>a lot that can be added. Some ideas:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L133" href="#L133">133</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L134" href="#L134">134</a><span class="code-text"><a-p>* Fix the benchmarks so that the resolver setup doesn&#39;t run every iteration.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L135" href="#L135">135</a><span class="code-text"><a-p>* Benchmark [this section of</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L136" href="#L136">136</a><span class="code-text"><a-p>  code](https://github.com/rust-lang/cargo/blob/a821e2cb24d7b6013433f069ab3bad53d160e100/src/cargo/ops/cargo_compile.rs#L470-L549)</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L137" href="#L137">137</a><span class="code-text"><a-p>  which builds the unit graph. The performance there isn&#39;t great, and it would</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L138" href="#L138">138</a><span class="code-text"><a-p>  be good to keep an eye on it. Unfortunately that would mean doing a bit of</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L139" href="#L139">139</a><span class="code-text"><a-p>  work to make `generate_targets` publicly visible, and there is a bunch of</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L140" href="#L140">140</a><span class="code-text"><a-p>  setup code that may need to be duplicated.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L141" href="#L141">141</a><span class="code-text"><a-p>* Benchmark the fingerprinting code.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L142" href="#L142">142</a><span class="code-text"><a-p>* Benchmark running the `cargo` executable. Running something like `cargo</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L143" href="#L143">143</a><span class="code-text"><a-p>  build` or `cargo check` with everything &quot;Fresh&quot; would be a good end-to-end</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L144" href="#L144">144</a><span class="code-text"><a-p>  exercise to measure the overall overhead of Cargo.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L145" href="#L145">145</a><span class="code-text"><a-p>* Benchmark pathological resolver scenarios. There might be some cases where</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L146" href="#L146">146</a><span class="code-text"><a-p>  the resolver can spend a significant amount of time. It would be good to</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L147" href="#L147">147</a><span class="code-text"><a-p>  identify if these exist, and create benchmarks for them. This may require</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L148" href="#L148">148</a><span class="code-text"><a-p>  creating an artificial index, similar to the `resolver-tests`. This should</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L149" href="#L149">149</a><span class="code-text"><a-p>  also consider scenarios where the resolver ultimately fails.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L150" href="#L150">150</a><span class="code-text"><a-p>* Benchmark without `Cargo.lock`. I&#39;m not sure if this is particularly</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L151" href="#L151">151</a><span class="code-text"><a-p>  valuable, since we are mostly concerned with incremental builds which will</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L152" href="#L152">152</a><span class="code-text"><a-p>  always have a lock file.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L153" href="#L153">153</a><span class="code-text"><a-p>* Benchmark just</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L154" href="#L154">154</a><span class="code-text"><a-p>  [`resolve::resolve`](https://github.com/rust-lang/cargo/blob/a821e2cb24d7b6013433f069ab3bad53d160e100/src/cargo/core/resolver/mod.rs#L122)</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L155" href="#L155">155</a><span class="code-text"><a-p>  without anything else. This can help focus on just the resolver.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L156" href="#L156">156</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L157" href="#L157">157</a><span class="code-text"><a-tu>[Criterion]</a-tu><a-p>: </a-p><a-tu>https://bheisler.github.io/criterion.rs/book/</a-tu></span></span></pre></div></div>
</main>
</body>
</html>
