<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Readme.md - fork/slpz @ all</title>
<meta name="description" content="Readme.md in fork/slpz at all.">
<meta property="og:title" content="Readme.md - fork/slpz @ all">
<meta property="og:description" content="Readme.md in fork/slpz at all.">
<meta property="og:type" content="website">
<meta property="og:site_name" content="git.julia.blue">
<link rel="stylesheet" href="/css/style.css?v=62">
<script type="module" src="/js/main.js?v=62"></script>
</head>
<body>
<main data-ref="all" data-tip="32f63687327542317516dfaac4196a1e204b4843">
<noscript>
  <div class="callout">
    <p>
      hey! this is a static git repo viewer. commit diffs, file history,
      older commits and other revisions won't load without javascript.
    </p>
    <p>
      you are still able to view source files on branch/tag tips,
      but the experience is degraded.
    </p>
  </div>
</noscript>
<header class="repo"><div>
<h1>fork/slpz</h1>
</div>
<pre class="clone">git clone https://git.julia.blue/fork/slpz</pre>
</header>
<p class="snapshot-commit"><span class="commit-author">Aitch</span><span class="commit-message" data-commit="afed6b7147e16d8f4fd686aaa7f9a40285e81044">make readme more readable for non-programmers</span><span class="snapshot-sha sha" data-commit="afed6b7147e16d8f4fd686aaa7f9a40285e81044">afed6b7</span><time datetime="2024-11-03T13:46:06-07:00" title="2024-11-03 13:46:06 -0700">2024-11-03</time></p>
<header class="topbar"><details class="switcher"><summary>all</summary><div class="menu"><strong>branches</strong><a class="current" href="/fork/slpz/ref/all/">all</a><a href="/fork/slpz/ref/main/">main</a><strong>tags</strong><a href="/fork/slpz/ref/v0.0/">v0.0</a><a class="all" href="/fork/slpz/refs">all refs →</a></div></details><nav class="crumbs"><a href="/fork/slpz/">slpz</a> / <span class="cur">Readme.md</span></nav>
<span class="view-stats"><span>4.1 KiB</span><span>89 lines</span></span><span class="actions"><a class="raw" href="/fork/slpz/raw/0a6b06b26eccb8c3666086acc4a095bf76451bcf/Readme.md">raw</a></span></header>
<div class="markdown-view"><input type="radio" name="markdown-view" id="markdown-rendered" checked><label for="markdown-rendered">rendered</label><input type="radio" name="markdown-view" id="markdown-code"><label for="markdown-code">code</label><section class="readme rendered"><h1>slpz</h1>
<p>Slpz compresses and decomoresses Slippi replay (slp) files.
The slpz format is 100% supported by <a href="https://x.com/rwing_aitch/status/1844056466283692388">rwing</a>, meaning you can watch back, browse, and export savestates even while your replays are compressed.</p>
<p>You can expect slpz files to be around 8x to 12x times smaller than slp files for regular matches. (~3Mb down to ~300Kb).
On my old thinkpad it can compress around 120 replays per second and decompress around 340 replays per second.</p>
<p>Compression is done with zstd.
zstd is not required on the user's computer.
The library is statically linked at compile time.</p>
<p>Important information, such as player tags, stages, date, characters, etc. all remain uncompressed in the slpz format.
This allows slp file browsers to easily parse and display this information without needing to decompress the replay.</p>
<h1>The slpz program</h1>
<p>You can download slpz <a href="https://github.com/AlexanderHarrison/slpz/releases/latest">here</a>.
This program allows commandline compression and decompression of both files and entire directories.</p>
<p>For example, the command <code>slpz -r --rm -x ~/Slippi/</code> will compress every replay in your Slippi replay directory.
The command <code>slpz -r --rm -d ~/Slippi/</code> will decompress them.</p>
<p><a href="https://x.com/rwing_aitch/status/1844056466283692388">Rwing</a> has a straightfoward process to compress/decompress built-in,
so you can use rwing if you don't want to use the command line.</p>
<p>Programmers can also use slpz as a <a href="https://crates.io/crates/slpz">library</a>.</p>
<h1>The slpz Format</h1>
<h2>Header</h2>
<p>24 bytes.</p>
<ul>
<li>0..4: Version. Current version is 0</li>
<li>4..8: Event Sizes offset</li>
<li>8..12: Game Start offset</li>
<li>12..16: Metadata offset</li>
<li>16..20: Compressed events offset</li>
<li>20..24: size of uncompressed events</li>
</ul>
<p>All offsets are from file start.</p>
<h2>Event Sizes</h2>
<p>This is equivalent to the 'Event Payloads' event in the <a href="https://github.com/project-slippi/slippi-wiki/blob/master/SPEC.md#event-payloads">SLP Spec</a>.</p>
<h2>Game start</h2>
<p>This is equivalent to the 'Game Start' event in the <a href="https://github.com/project-slippi/slippi-wiki/blob/master/SPEC.md#game-start">SLP Spec</a>.</p>
<h2>Metadata</h2>
<p>This is equivalent to the 'Metadata' event in the <a href="https://github.com/project-slippi/slippi-wiki/blob/master/SPEC.md#the-metadata-element">SLP Spec</a>.</p>
<h2>Compressed Events</h2>
<p>This is reordered events passed through zstd compression.</p>
<h3>Event Reordering?</h3>
<p>Reordering the bytes in events increases the compression ratio ~2x.</p>
<p>A normal slp file is a stream of events consisting of a command byte and statically sized payload.
Event payloads are almost all the same, so we can reorder the data to increase the compressability of the data.</p>
<p>We first turn the event stream into a list of command bytes, keeping the order but removing the payloads.
Then a list of all of the first bytes in the payloads for events with command 0,
then all of the second bytes in the payloads for events with command 0,
all the way to a list of the last bytes of the payloads for command 255.</p>
<p>To undo this reordering we also need the number of total events, so we put this in as 4 bytes at the start.</p>
<h4>Example</h4>
<pre>cmd ABCD cmd2 EFG cmd ABCD cmd3 HI cmd2 EFG
</pre>
<p>converts to:</p>
<pre>// Event Order
5 cmd cmd2 cmd cmd3 cmd2
// Reordered Event Data
AABBCCDD EEFFGG HI
</pre>
<h1>Comparison with <a href="https://github.com/pcrain/slippc">slippc</a></h1>
<p>slippc is very impressive.
They have achieved much higher compression rates by abusing the contents of events.
However, in my opinion, this comes with two big drawbacks:</p>
<ol>
<li><strong>Maintentance</strong>: Due to abusing the structure of events, slippc is beholden to the slp spec and must be manually updated for each new version.
<em>Slippc has not been updated for over a year and fails on new replays.</em>
slpz does not care about the contents of events. (Other than the Event Payloads event).
It will work for all slp spec changes in the future.</li>
<li><strong>Performance</strong>: slpz uses zstd compression. slippc uses lzma compression.
lzma compresses slightly better than zstd, but takes order of magnitudes longer to compress and decompress.
Incredibly fast decompression allows slpz files to be watched back, browsed, and used just like regular slp files,
without needing seconds of waiting for decompression.</li>
</ol>
</section><div class="code-panel"><pre class="code src language-markdown"><span class="code-line"><a class="ln" id="L1" href="#L1">1</a><span class="code-text"><a-p>#</a-p> <a-tt>slpz</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L2" href="#L2">2</a><span class="code-text">Slpz compresses and decomoresses Slippi replay (slp) files. </span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L3" href="#L3">3</a><span class="code-text">The slpz format is 100% supported by [rwing](https://x.com/rwing_aitch/status/1844056466283692388), meaning you can watch back, browse, and export savestates even while your replays are compressed.  </span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L4" href="#L4">4</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L5" href="#L5">5</a><span class="code-text">You can expect slpz files to be around 8x to 12x times smaller than slp files for regular matches. (~3Mb down to ~300Kb).</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L6" href="#L6">6</a><span class="code-text">On my old thinkpad it can compress around 120 replays per second and decompress around 340 replays per second.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L7" href="#L7">7</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L8" href="#L8">8</a><span class="code-text">Compression is done with zstd.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L9" href="#L9">9</a><span class="code-text">zstd is not required on the user&#39;s computer.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L10" href="#L10">10</a><span class="code-text">The library is statically linked at compile time.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L11" href="#L11">11</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L12" href="#L12">12</a><span class="code-text">Important information, such as player tags, stages, date, characters, etc. all remain uncompressed in the slpz format. </span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L13" href="#L13">13</a><span class="code-text">This allows slp file browsers to easily parse and display this information without needing to decompress the replay.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L14" href="#L14">14</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L15" href="#L15">15</a><span class="code-text"><a-p>#</a-p> <a-tt>The slpz program</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L16" href="#L16">16</a><span class="code-text">You can download slpz [here](https://github.com/AlexanderHarrison/slpz/releases/latest).</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L17" href="#L17">17</a><span class="code-text">This program allows commandline compression and decompression of both files and entire directories.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L18" href="#L18">18</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L19" href="#L19">19</a><span class="code-text">For example, the command `slpz -r --rm -x ~/Slippi/` will compress every replay in your Slippi replay directory.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L20" href="#L20">20</a><span class="code-text">The command `slpz -r --rm -d ~/Slippi/` will decompress them.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L21" href="#L21">21</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L22" href="#L22">22</a><span class="code-text">[Rwing](https://x.com/rwing_aitch/status/1844056466283692388) has a straightfoward process to compress/decompress built-in, </span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L23" href="#L23">23</a><span class="code-text">so you can use rwing if you don&#39;t want to use the command line.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L24" href="#L24">24</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L25" href="#L25">25</a><span class="code-text">Programmers can also use slpz as a [library](https://crates.io/crates/slpz).</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L26" href="#L26">26</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L27" href="#L27">27</a><span class="code-text"><a-p>#</a-p> <a-tt>The slpz Format</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L28" href="#L28">28</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L29" href="#L29">29</a><span class="code-text"><a-p>##</a-p> <a-tt>Header</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L30" href="#L30">30</a><span class="code-text">24 bytes.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L31" href="#L31">31</a><span class="code-text"><a-p>- </a-p>0..4: Version. Current version is 0</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L32" href="#L32">32</a><span class="code-text"><a-p>- </a-p>4..8: Event Sizes offset</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L33" href="#L33">33</a><span class="code-text"><a-p>- </a-p>8..12: Game Start offset</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L34" href="#L34">34</a><span class="code-text"><a-p>- </a-p>12..16: Metadata offset</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L35" href="#L35">35</a><span class="code-text"><a-p>- </a-p>16..20: Compressed events offset</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L36" href="#L36">36</a><span class="code-text"><a-p>- </a-p>20..24: size of uncompressed events</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L37" href="#L37">37</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L38" href="#L38">38</a><span class="code-text"><a-p>All offsets are from file start.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L39" href="#L39">39</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L40" href="#L40">40</a><span class="code-text"><a-p>## </a-p><a-tt>Event Sizes</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L41" href="#L41">41</a><span class="code-text"><a-p>This is equivalent to the &#39;Event Payloads&#39; event in the [SLP Spec](https://github.com/project-slippi/slippi-wiki/blob/master/SPEC.md#event-payloads).</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L42" href="#L42">42</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L43" href="#L43">43</a><span class="code-text"><a-p>## </a-p><a-tt>Game start</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L44" href="#L44">44</a><span class="code-text"><a-p>This is equivalent to the &#39;Game Start&#39; event in the [SLP Spec](https://github.com/project-slippi/slippi-wiki/blob/master/SPEC.md#game-start).</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L45" href="#L45">45</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L46" href="#L46">46</a><span class="code-text"><a-p>## </a-p><a-tt>Metadata</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L47" href="#L47">47</a><span class="code-text"><a-p>This is equivalent to the &#39;Metadata&#39; event in the [SLP Spec](https://github.com/project-slippi/slippi-wiki/blob/master/SPEC.md#the-metadata-element).</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L48" href="#L48">48</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L49" href="#L49">49</a><span class="code-text"><a-p>## </a-p><a-tt>Compressed Events</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L50" href="#L50">50</a><span class="code-text"><a-p>This is reordered events passed through zstd compression.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L51" href="#L51">51</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L52" href="#L52">52</a><span class="code-text"><a-p>### </a-p><a-tt>Event Reordering?</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L53" href="#L53">53</a><span class="code-text"><a-p>Reordering the bytes in events increases the compression ratio ~2x.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L54" href="#L54">54</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L55" href="#L55">55</a><span class="code-text"><a-p>A normal slp file is a stream of events consisting of a command byte and statically sized payload.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L56" href="#L56">56</a><span class="code-text"><a-p>Event payloads are almost all the same, so we can reorder the data to increase the compressability of the data.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L57" href="#L57">57</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L58" href="#L58">58</a><span class="code-text"><a-p>We first turn the event stream into a list of command bytes, keeping the order but removing the payloads.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L59" href="#L59">59</a><span class="code-text"><a-p>Then a list of all of the first bytes in the payloads for events with command 0, </a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L60" href="#L60">60</a><span class="code-text"><a-p>then all of the second bytes in the payloads for events with command 0,</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L61" href="#L61">61</a><span class="code-text"><a-p>all the way to a list of the last bytes of the payloads for command 255.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L62" href="#L62">62</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L63" href="#L63">63</a><span class="code-text"><a-p>To undo this reordering we also need the number of total events, so we put this in as 4 bytes at the start.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L64" href="#L64">64</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L65" href="#L65">65</a><span class="code-text"><a-p>#### </a-p><a-tt>Example</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L66" href="#L66">66</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L67" href="#L67">67</a><span class="code-text"><a-p>cmd ABCD cmd2 EFG cmd ABCD cmd3 HI cmd2 EFG</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L68" href="#L68">68</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L69" href="#L69">69</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L70" href="#L70">70</a><span class="code-text"><a-p>converts to:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L71" href="#L71">71</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L72" href="#L72">72</a><span class="code-text"><a-p>// Event Order</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L73" href="#L73">73</a><span class="code-text"><a-p>5 cmd cmd2 cmd cmd3 cmd2</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L74" href="#L74">74</a><span class="code-text"><a-p>// Reordered Event Data</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L75" href="#L75">75</a><span class="code-text"><a-p>AABBCCDD EEFFGG HI</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L76" href="#L76">76</a><span class="code-text"><a-p>```</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L77" href="#L77">77</a><span class="code-text"></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L78" href="#L78">78</a><span class="code-text"><a-p># </a-p><a-tt>Comparison with [slippc](https://github.com/pcrain/slippc)</a-tt></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L79" href="#L79">79</a><span class="code-text"><a-p>slippc is very impressive. </a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L80" href="#L80">80</a><span class="code-text"><a-p>They have achieved much higher compression rates by abusing the contents of events.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L81" href="#L81">81</a><span class="code-text"><a-p>However, in my opinion, this comes with two big drawbacks:</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L82" href="#L82">82</a><span class="code-text"><a-p>1. **Maintentance**: Due to abusing the structure of events, slippc is beholden to the slp spec and must be manually updated for each new version.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L83" href="#L83">83</a><span class="code-text"><a-p>*Slippc has not been updated for over a year and fails on new replays.*</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L84" href="#L84">84</a><span class="code-text"><a-p>slpz does not care about the contents of events. (Other than the Event Payloads event). </a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L85" href="#L85">85</a><span class="code-text"><a-p>It will work for all slp spec changes in the future.</a-p></span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L86" href="#L86">86</a><span class="code-text"><a-p>2. </a-p>**Performance**: slpz uses zstd compression. slippc uses lzma compression.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L87" href="#L87">87</a><span class="code-text">lzma compresses slightly better than zstd, but takes order of magnitudes longer to compress and decompress.</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L88" href="#L88">88</a><span class="code-text">Incredibly fast decompression allows slpz files to be watched back, browsed, and used just like regular slp files,</span><span class="code-break">
</span></span><span class="code-line"><a class="ln" id="L89" href="#L89">89</a><span class="code-text">without needing seconds of waiting for decompression.</span></span></pre></div></div>
</main>
</body>
</html>
