Tianon's Ramblings2024-03-15T16:19:55-06:00https://tianon.github.ioTianon Graviadmwiggin@gmail.comhttps://tianon.github.iohttps://tianon.github.io/post/2021/03/16/docker-setup-rereduxMy Docker Install Process (re-redux)2021-03-16T00:00:00-06:002021-03-16T00:00:00-06:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>See <a href="/post/2016/12/07/docker-setup.html">“My Docker Install Process”</a> and <a href="/post/2017/05/18/docker-setup-redux.html">“My Docker Install Process (redux)”</a>. This one’s going to be even more to-the-point.</p>
<h1 id="grab-dockers-apt-repo-gpg-key">grab Docker’s APT repo GPG key</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">GNUPGHOME</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">mktemp</span> <span class="nt">-d</span><span class="si">)</span><span class="s2">"</span><span class="p">;</span> <span class="nb">export </span>GNUPGHOME
gpg <span class="nt">--keyserver</span> keyserver.ubuntu.com <span class="nt">--recv-keys</span> 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
<span class="nb">sudo mkdir</span> <span class="nt">-p</span> /etc/apt/tianon.gpg.d
gpg <span class="nt">--export</span> <span class="nt">--armor</span> 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 | <span class="nb">sudo tee</span> /etc/apt/tianon.gpg.d/docker.gpg.asc
<span class="nb">rm</span> <span class="nt">-rf</span> <span class="s2">"</span><span class="nv">$GNUPGHOME</span><span class="s2">"</span>
</code></pre></div></div>
<h1 id="add-dockers-apt-source">add Docker’s APT source</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">source</span> /etc/os-release
<span class="nb">echo</span> <span class="s2">"deb [ arch=amd64 signed-by=/etc/apt/tianon.gpg.d/docker.gpg.asc ] https://download.docker.com/linux/debian </span><span class="nv">$VERSION_CODENAME</span><span class="s2"> stable"</span> | <span class="nb">sudo tee</span> /etc/apt/sources.list.d/docker.list
</code></pre></div></div>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt update
<span class="c">...
</span><span class="go">Get:6 https://download.docker.com/linux/debian buster/stable amd64 Packages [17.8 kB]
</span><span class="c">...
</span><span class="go">Reading package lists... Done
</span></code></pre></div></div>
<h1 id="exclude-unwated-cli-plugins">exclude (unwated) CLI plugins</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s1">'path-exclude /usr/libexec/docker/cli-plugins/*'</span> | <span class="nb">sudo tee</span> /etc/dpkg/dpkg.cfg.d/unwanted-docker-cli-plugins
</code></pre></div></div>
<h1 id="pin-docker-versions">pin Docker versions</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>vim /etc/apt/preferences.d/docker.pref
</code></pre></div></div>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">Package: *aufs* *rootless* cgroupfs-mount docker-*-plugin
Pin: version *
Pin-Priority: -10
Package: docker*
Pin: version 5:20.10*
Pin-Priority: 999
Package: containerd*
Pin: version 1.4*
Pin-Priority: 999
</span></code></pre></div></div>
<h1 id="pre-configure-docker">pre-configure Docker</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /etc/docker
<span class="nb">sudo </span>vim /etc/docker/daemon.json
</code></pre></div></div>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"storage-driver"</span><span class="p">:</span><span class="w"> </span><span class="s2">"overlay2"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h1 id="configure-boot-parameters">configure boot parameters</h1>
<blockquote>
<p>I usually set a few boot parameters as well (in <code class="language-plaintext highlighter-rouge">/etc/default/grub</code>’s <code class="language-plaintext highlighter-rouge">GRUB_CMDLINE_LINUX_DEFAULT</code> option – run <code class="language-plaintext highlighter-rouge">sudo update-grub</code> after adding these, space-separated).</p>
</blockquote>
<ul>
<li><code class="language-plaintext highlighter-rouge">cgroup_enable=memory</code> – enable “memory accounting” for containers (allows <code class="language-plaintext highlighter-rouge">docker run --memory</code> for setting hard memory limits on containers)</li>
<li><code class="language-plaintext highlighter-rouge">swapaccount=1</code> – enable “swap accounting” for containers (allows <code class="language-plaintext highlighter-rouge">docker run --memory-swap</code> for setting hard swap memory limits on containers)</li>
<li><code class="language-plaintext highlighter-rouge">vsyscall=emulate</code> – allow older binaries to run (<code class="language-plaintext highlighter-rouge">debian:wheezy</code>, etc.; see <a href="https://github.com/docker/docker/issues/28705">docker/docker#28705</a>)</li>
<li><code class="language-plaintext highlighter-rouge">systemd.legacy_systemd_cgroup_controller=yes</code> – newer versions of systemd <em>may</em> disable the legacy cgroup interfaces Docker currently uses; this instructs systemd to keep those enabled (for more details, see <a href="https://github.com/systemd/systemd/pull/4628">systemd/systemd#4628</a>, <a href="https://github.com/opencontainers/runc/issues/1175">opencontainers/runc#1175</a>, <a href="https://github.com/docker/docker/issues/28109">docker/docker#28109</a>)
<ul>
<li>NOTE: this one gets more complicated in Debian 11+ (“Bullseye”); possibly worth switching to cgroupv2 and <code class="language-plaintext highlighter-rouge">systemd.unified_cgroup_hierarchy=1</code></li>
</ul>
</li>
</ul>
<p>All together:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
<span class="nv">GRUB_CMDLINE_LINUX_DEFAULT</span><span class="o">=</span><span class="s2">"cgroup_enable=memory swapaccount=1 vsyscall=emulate systemd.legacy_systemd_cgroup_controller=yes"</span>
...
</code></pre></div></div>
<p>(Don’t forget to <code class="language-plaintext highlighter-rouge">sudo update-grub</code> and potentially reboot – check <code class="language-plaintext highlighter-rouge">/proc/cmdline</code> to verify.)</p>
<h1 id="install-docker">install Docker!</h1>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install</span> <span class="nt">-V</span> docker-ce
<span class="c">...
</span><span class="go">Unpacking containerd.io (1.4.4-1) ...
</span><span class="c">...
</span><span class="go">Unpacking docker-ce-cli (5:20.10.5~3-0~debian-buster) ...
</span><span class="c">...
</span><span class="go">Unpacking docker-ce (5:20.10.5~3-0~debian-buster) ...
</span><span class="c">...
</span><span class="go">
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>usermod <span class="nt">-aG</span> docker <span class="s2">"</span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-un</span><span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>
https://tianon.github.io/post/2018/01/11/iscsiiSCSI in Debian2018-01-11T00:00:00-07:002018-01-11T00:00:00-07:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>I’ve recently been playing with Debian’s iSCSI support, and it’s pretty neat.</p>
<p>It was a little esoteric to set things up, so I figured I’d write up a quick blog post of exactly what I did both for my own future-self’s sake and for the sake of anyone else trying to do something similar.</p>
<p>The most “followable” guide I found was <a href="https://www.certdepot.net/rhel7-configure-iscsi-target-initiator-persistently/">https://www.certdepot.net/rhel7-configure-iscsi-target-initiator-persistently/</a> (which the below is probably really similar to).</p>
<p>The exact details of what I was trying to accomplish are as follows:</p>
<ul>
<li>100GB “sparse” file on <code class="language-plaintext highlighter-rouge">my-desktop</code></li>
<li>presented as an iSCSI target</li>
<li>mounted on <code class="language-plaintext highlighter-rouge">my-rpi3</code> as <code class="language-plaintext highlighter-rouge">/var/lib/docker</code> (preferably with <code class="language-plaintext highlighter-rouge">discard</code> enabled so the file on <code class="language-plaintext highlighter-rouge">my-desktop</code> stays sparse)</li>
</ul>
<p>On <code class="language-plaintext highlighter-rouge">my-desktop</code>, I used the <code class="language-plaintext highlighter-rouge">targetcli-fb</code> package to configure my iSCSI target:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt <span class="nb">install </span>targetcli-fb
<span class="go">
</span><span class="gp">$</span><span class="w"> </span><span class="c"># create the sparse file</span>
<span class="gp">$</span><span class="w"> </span><span class="nb">mkdir</span> <span class="nt">-p</span> /home/tianon/iscsi
<span class="gp">$</span><span class="w"> </span><span class="nb">truncate</span> <span class="nt">--size</span><span class="o">=</span>100G /home/tianon/iscsi/my-rpi3-docker.img
<span class="go">
</span><span class="gp">$</span><span class="w"> </span><span class="c"># launch "targetcli" to configure the iSCSI bits</span>
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>targetcli
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>create a <span class="s2">"fileio"</span> object connected to the new sparse file
<span class="gp">/></span><span class="w"> </span>/backstores/fileio create <span class="nv">name</span><span class="o">=</span>my-rpi3-docker <span class="nv">file_or_dev</span><span class="o">=</span>/home/tianon/iscsi/my-rpi3-docker.img
<span class="go">
</span><span class="gp">#</span><span class="w"> </span><span class="nb">enable</span> <span class="s2">"emulated TPU"</span> <span class="o">(</span><span class="nb">enable </span>TRIM / UNMAP / DISCARD<span class="o">)</span>
<span class="gp">/></span><span class="w"> </span>/backstores/fileio/my-rpi3-docker <span class="nb">set </span>attribute <span class="nv">emulate_tpu</span><span class="o">=</span>1
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>create iSCSI storage object
<span class="gp">/></span><span class="w"> </span>/iscsi create iqn.1992-01.com.example.my-desktop:storage:my-rpi3-docker
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>create <span class="s2">"LUN"</span> assigned to the <span class="s2">"fileio"</span> object
<span class="gp">/></span><span class="w"> </span>/iscsi/iqn.1992-01.com.example.my-desktop:storage:my-rpi3-docker/tpg1/luns create /backstores/fileio/my-rpi3-docker
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>create an ACL <span class="k">for </span>my-rpi3 to connect
<span class="gp">/></span><span class="w"> </span>/iscsi/iqn.1992-01.com.example.my-desktop:storage:my-rpi3-docker/tpg1/acls create iqn.1992-01.com.example:node:my-rpi3
<span class="gp">#</span><span class="w"> </span>and <span class="nb">set </span>a CHAP username and password, <span class="k">for </span>security
<span class="gp">/></span><span class="w"> </span>/iscsi/iqn.1992-01.com.example.my-desktop:storage:my-rpi3-docker/tpg1/acls/iqn.1992-01.com.example:node:my-rpi3 <span class="nb">set </span>auth <span class="nv">userid</span><span class="o">=</span>rpi3 <span class="nv">password</span><span class="o">=</span>holy-cow-this-iscsi-password-is-so-secret-nobody-will-evvvvvvvvver-guess-it
</code></pre></div></div>
<p>Additionally, I’ve been experimenting with <code class="language-plaintext highlighter-rouge">firewalld</code> on <code class="language-plaintext highlighter-rouge">my-desktop</code>, so I had to add the <code class="language-plaintext highlighter-rouge">iscsi-target</code> service to my <code class="language-plaintext highlighter-rouge">internal</code> zone to allow the traffic from <code class="language-plaintext highlighter-rouge">my-rpi3</code>.</p>
<p>On <code class="language-plaintext highlighter-rouge">my-rpi3</code>, I used the <code class="language-plaintext highlighter-rouge">open-iscsi</code> package to configure my iSCSI initiator:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt <span class="nb">install </span>open-iscsi
<span class="go">
</span><span class="gp">$</span><span class="w"> </span><span class="c"># update "InitiatorName" to match the value from our ACL above</span>
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>vim /etc/iscsi/initiatorname.iscsi
<span class="go">InitiatorName=iqn.1992-01.com.example:node:my-rpi3
</span><span class="gp">$</span><span class="w"> </span><span class="c"># update "node.startup" and "node.session.auth.*" for our CHAP credentials from above</span>
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>vim /etc/iscsi/iscsid.conf
<span class="c">...
</span><span class="go">node.startup = automatic
</span><span class="c">...
</span><span class="go">node.session.auth.authmethod = CHAP
node.session.auth.username = rpi3
node.session.auth.password = holy-cow-this-iscsi-password-is-so-secret-nobody-will-evvvvvvvvver-guess-it
</span><span class="c">...
</span><span class="go">
</span><span class="gp">#</span><span class="w"> </span>restart iscsid so all that takes effect <span class="o">(</span>especially the InitiatorName change<span class="o">)</span>
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>systemctl restart iscsid
<span class="go">
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>iscsiadm <span class="nt">--mode</span> discovery <span class="nt">--type</span> sendtargets <span class="nt">--portal</span> my-desktop-ip-address
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>iscsiadm <span class="nt">--mode</span> node <span class="nt">--targetname</span> iqn.1992-01.com.example.my-desktop:storage:my-rpi3-docker <span class="nt">--portal</span> my-desktop-ip-address <span class="nt">--login</span>
<span class="go">
</span><span class="gp">$</span><span class="w"> </span>lsblk <span class="nt">--scsi</span>
<span class="go">NAME HCTL TYPE VENDOR MODEL REV TRAN
sda 0:0:0:0 disk LIO-ORG my-rpi3-docker 4.0 iscsi
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>fdisk /dev/sda
<span class="c">...
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>mkfs.ext4 <span class="nt">-T</span> news <span class="nt">-L</span> my-rpi3-docker /dev/sda1
<span class="c">...
</span><span class="gp">$</span><span class="w"> </span>lsblk | <span class="nb">grep </span>my-rpi3-docker
<span class="go">... UUID="xxx" ...
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>vim /etc/fstab
<span class="c">...
</span><span class="go">UUID="xxx" /var/lib/docker ext4 noatime,discard,_netdev 0 0
</span><span class="c">...
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>systemctl stop docker
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>mount /var/lib/docker
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>systemctl start docker
<span class="go">
</span><span class="gp">$</span><span class="w"> </span><span class="c"># yay, profit (and should auto-remount properly on boot and everything, too)</span>
</code></pre></div></div>
<p>(Obviously, replace <code class="language-plaintext highlighter-rouge">iqn.1992-01.com.example</code> with an appropriate IQN for your own domain <a href="https://en.wikipedia.org/wiki/ISCSI#Addressing">as described on Wikipedia</a>, and other values as appropriate like the username/password, hostnames, IPs, etc.)</p>
<p>As for speed, I was able to get the following result from a very simplified <code class="language-plaintext highlighter-rouge">dd</code>-based speed test – YMMV:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">dd </span><span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">of</span><span class="o">=</span>/var/lib/docker/testfile <span class="nv">bs</span><span class="o">=</span>100M <span class="nv">count</span><span class="o">=</span>10 <span class="nv">oflag</span><span class="o">=</span>direct
<span class="go">10+0 records in
10+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 97.9608 s, 10.7 MB/s
</span></code></pre></div></div>
https://tianon.github.io/post/2017/12/26/dockerize-compiled-softwareDockerizing Compiled Software2017-12-26T00:00:00-07:002017-12-26T00:00:00-07:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>I recently went through a stint of closing a <em>huge</em> number of issues in the <a href="https://github.com/docker-library/php">docker-library/php</a> repository, and one of the oldest (and longest) discussions was related to installing dependencies for a compiling extensions, and I wrote a <a href="https://github.com/docker-library/php/issues/75#issuecomment-353673374">semi-long comment</a> explaining how I do this in a general way for any software I wish to Dockerize.</p>
<p>I’m going to copy most of that comment here and perhaps expand a little bit more in order to have a better/cleaner place to link to!</p>
<p>The first step I take is to write the naïve version of the <code class="language-plaintext highlighter-rouge">Dockerfile</code>: download the source, run <code class="language-plaintext highlighter-rouge">./configure && make</code> etc, clean up. I then try building my naïve creation, and in doing so hope for an error message. (yes, really!)</p>
<p>The error message will usually take the form of something like <code class="language-plaintext highlighter-rouge">error: could not find "xyz.h"</code> or <code class="language-plaintext highlighter-rouge">error: libxyz development headers not found</code>.</p>
<p>If I’m building in Debian, I’ll hit <code class="language-plaintext highlighter-rouge">https://packages.debian.org/file:xyz.h</code> (replacing “xyz.h” with the name of the header file from the error message), or even just Google something like “xyz.h debian”, to figure out the name of the package I require.</p>
<p>If I’m building in Alpine, I’ll use <a href="https://pkgs.alpinelinux.org/contents">https://pkgs.alpinelinux.org/contents</a> to perform a similar search.</p>
<p>The same works to some extent for “libxyz development headers”, but in my experience Google works better for those since different distributions and projects will call these development packages by different names, so sometimes it’s a little harder to figure out exactly which one is the “right” one to install.</p>
<p>Once I’ve got a package name, I add that package name to my <code class="language-plaintext highlighter-rouge">Dockerfile</code>, rinse, and repeat. Eventually, this usually leads to a successful build. Occasionally I find that some library either isn’t in Debian or Alpine, or isn’t new enough, and I’ve also got to build it from source, but those instances are rare in my own experience – YMMV.</p>
<p>I’ll also often check the source for the Debian (via <a href="https://sources.debian.org">https://sources.debian.org</a>) or Alpine (via <a href="https://git.alpinelinux.org/cgit/aports/tree">https://git.alpinelinux.org/cgit/aports/tree</a>) package of the software I’m looking to compile, especially paying attention to <code class="language-plaintext highlighter-rouge">Build-Depends</code> (ala <a href="https://sources.debian.org/src/php7.0/7.0.26-1/debian/control/"><code class="language-plaintext highlighter-rouge">php7.0=7.0.26-1</code>’s <code class="language-plaintext highlighter-rouge">debian/control</code> file</a>) and/or <code class="language-plaintext highlighter-rouge">makedepends</code> (ala <a href="https://git.alpinelinux.org/cgit/aports/tree/community/php7/APKBUILD?id=d0ca197f031f96d4664cafaa618aeccf94640a1e"><code class="language-plaintext highlighter-rouge">php7</code>’s <code class="language-plaintext highlighter-rouge">APKBUILD</code> file</a>) for package name clues.</p>
<p>Personally, I find this sort of detective work interesting and rewarding, but I realize I’m probably a bit of a unique creature. Another good technique I use occasionally is to determine whether anyone else has already Dockerized the thing I’m trying to, so I can simply learn directly from their <code class="language-plaintext highlighter-rouge">Dockerfile</code> which packages I’ll need to install.</p>
<p>For the specific case of PHP extensions, there’s almost always someone who’s already figured out what’s necessary for this or that module, and all I have to do is some light detective work to find them.</p>
<p>Anyways, that’s my method! Hope it’s helpful, and happy hunting!</p>
https://tianon.github.io/post/2017/05/23/debuerreotypeDebuerreotype2017-05-23T00:00:00-06:002017-05-23T00:00:00-06:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>Following in the footsteps of one of my favorite Debian Developers, <a href="https://github.com/lamby">Chris Lamb / lamby</a> (who is <a href="https://bugs.debian.org/from:lamby@debian.org">quite prolific</a> in the reproducible builds effort within Debian), I’ve started a new project based on <a href="http://snapshot.debian.org">snapshot.debian.org (time-based snapshots of the Debian archive)</a> and some of <a href="https://github.com/lamby/debootstrap/commit/66b15380814aa62ca4b5807270ac57a3c8a0558d">lamby’s work</a> for creating reproducible Debian (<a href="https://www.debian.org/releases/stretch/amd64/apds03.html.en#idp54701872"><code class="language-plaintext highlighter-rouge">debootstrap</code></a>) rootfs tarballs.</p>
<p>The project is named <a href="https://github.com/debuerreotype/debuerreotype">“Debuerreotype”</a> as an homage to the photography roots of the word “snapshot” and the <a href="https://en.wikipedia.org/wiki/Daguerreotype">daguerreotype</a> process which was an early method of taking photographs. The essential goal is to create “photographs” of a minimal Debian rootfs, so the name seemed appropriate (even if it’s a bit on the “mouthful” side).</p>
<p>The end-goal is to create and release Debian rootfs tarballs for a given point-in-time (especially for use in Docker) which should be fully reproducible, and thus improve confidence in the provenance of the <a href="https://hub.docker.com/_/debian/">Debian Docker base images</a>.</p>
<p>For more information about reproducibility and why it matters, see <a href="https://reproducible-builds.org/">reproducible-builds.org</a>, which has more thorough explanations of the why and how and links to other important work such as the <a href="https://tests.reproducible-builds.org/debian/">reproducible builds effort in Debian (for Debian package builds)</a>.</p>
<p>In order to verify that the tool actually works as intended, I ran builds against seven explicit architectures (<code class="language-plaintext highlighter-rouge">amd64</code>, <code class="language-plaintext highlighter-rouge">arm64</code>, <code class="language-plaintext highlighter-rouge">armel</code>, <code class="language-plaintext highlighter-rouge">armhf</code>, <code class="language-plaintext highlighter-rouge">i386</code>, <code class="language-plaintext highlighter-rouge">ppc64el</code>, <code class="language-plaintext highlighter-rouge">s390x</code>) and eight explicit suites (<code class="language-plaintext highlighter-rouge">oldstable</code>, <code class="language-plaintext highlighter-rouge">stable</code>, <code class="language-plaintext highlighter-rouge">testing</code>, <code class="language-plaintext highlighter-rouge">unstable</code>, <code class="language-plaintext highlighter-rouge">wheezy</code>, <code class="language-plaintext highlighter-rouge">jessie</code>, <code class="language-plaintext highlighter-rouge">stretch</code>, <code class="language-plaintext highlighter-rouge">sid</code>).</p>
<p>I used a timestamp value of <code class="language-plaintext highlighter-rouge">2017-05-16T00:00:00Z</code>, and skipped combinations that don’t exist (such as <code class="language-plaintext highlighter-rouge">wheezy</code> on <code class="language-plaintext highlighter-rouge">arm64</code>) or aren’t supported anymore (such as <code class="language-plaintext highlighter-rouge">wheezy</code> on <code class="language-plaintext highlighter-rouge">s390x</code>). I ran the scripts repeatedly over several days, using <a href="https://diffoscope.org/"><code class="language-plaintext highlighter-rouge">diffoscope</code></a> to compare the results.</p>
<p>While doing said testing, I ran across <a href="https://bugs.debian.org/857803">#857803</a>, and <a href="https://github.com/debuerreotype/debuerreotype/commit/c90f2e5e6c319c31f9668cec10e93b86b46d9417#diff-70efd6067d981af974e9424ee04ca8b6">added a workaround</a>. There’s also <a href="https://github.com/debuerreotype/debuerreotype/blob/e208ee09d83f1101aa378aa6e5a697e8ee3f0cbc/README.md#why-isnt-wheezy-reproducible">a minor outstanding issue with <code class="language-plaintext highlighter-rouge">wheezy</code>’s reproducibility</a> that I haven’t had a chance to dig deep very deeply into yet (but it’s pretty benign and Wheezy’s LTS support window ends <a href="https://wiki.debian.org/LTS">2018-05-31</a>, so I’m not too stressed about it).</p>
<p>I’ve also packaged the tool for Debian, and submitted it into the <a href="https://ftp-master.debian.org/new.html">NEW queue</a>, so hopefully the <a href="https://ftp-master.debian.org/">FTP Masters</a> will look favorably upon this being a tool that’s available to install from the Debian archive as well. 😇</p>
<p>Anyhow, please <a href="https://github.com/debuerreotype/debuerreotype/blob/e208ee09d83f1101aa378aa6e5a697e8ee3f0cbc/README.md#usage">give it a try</a>, have fun, and as always, <a href="https://github.com/debuerreotype/debuerreotype/issues">report bugs</a>!</p>
https://tianon.github.io/post/2017/05/18/docker-setup-reduxMy Docker Install Process (redux)2017-05-18T00:00:00-06:002017-05-18T00:00:00-06:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>Since I wrote <a href="/post/2016/12/07/docker-setup.html">my first post on this topic</a>, Docker has switched from <a href="https://apt.dockerproject.org/repo">apt.dockerproject.org</a> to <a href="https://download.docker.com/linux/debian">download.docker.com</a>, so this post revisits my original steps, but tailored for the new repo.</p>
<p>There will be less commentary this time (straight to the beef). For further commentary on “why” for any step, see my previous post.</p>
<blockquote>
<p>These steps <em>should</em> be fairly similar to what’s found in <a href="https://docs.docker.com/engine/installation/linux/debian/">upstream’s “Install Docker on Debian” document</a>, but do differ slightly in a few minor ways.</p>
</blockquote>
<h1 id="grab-dockers-apt-repo-gpg-key">grab Docker’s APT repo GPG key</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># "Docker Release (CE deb)"</span>
<span class="nb">export </span><span class="nv">GNUPGHOME</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">mktemp</span> <span class="nt">-d</span><span class="si">)</span><span class="s2">"</span>
gpg <span class="nt">--keyserver</span> keyserver.ubuntu.com <span class="nt">--recv-keys</span> 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
<span class="c"># stretch+</span>
gpg <span class="nt">--export</span> <span class="nt">--armor</span> 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 | <span class="nb">sudo tee</span> /etc/apt/trusted.gpg.d/docker.gpg.asc
<span class="c"># jessie</span>
<span class="c"># gpg --export 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 | sudo tee /etc/apt/trusted.gpg.d/docker.gpg > /dev/null</span>
<span class="nb">rm</span> <span class="nt">-rf</span> <span class="s2">"</span><span class="nv">$GNUPGHOME</span><span class="s2">"</span>
</code></pre></div></div>
<p>(<strong>Update 2017-09-29:</strong> If you’re installing EE, the key changes to <code class="language-plaintext highlighter-rouge">DD911E995A64A202E85907D6BC14F10B6D085F96</code>.)</p>
<p>Verify:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>apt-key list
<span class="c">...
</span><span class="go">
/etc/apt/trusted.gpg.d/docker.gpg.asc
-------------------------------------
pub rsa4096 2017-02-22 [SCEA]
9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88
</span><span class="gp">uid [ unknown] Docker Release (CE deb) <docker@docker.com></span><span class="w">
</span><span class="go">sub rsa4096 2017-02-22 [S]
</span><span class="c">...
</span></code></pre></div></div>
<h1 id="add-dockers-apt-source">add Docker’s APT source</h1>
<p>With the switch to download.docker.com, HTTPS is now mandated:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>apt-get update <span class="o">&&</span> apt-get <span class="nb">install </span>apt-transport-https
</code></pre></div></div>
<p>Setup <code class="language-plaintext highlighter-rouge">sources.list</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"deb [ arch=amd64 ] https://download.docker.com/linux/debian stretch stable"</span> | <span class="nb">sudo tee</span> /etc/apt/sources.list.d/docker.list
</code></pre></div></div>
<p>Add <code class="language-plaintext highlighter-rouge">edge</code> component for every-month releases and <code class="language-plaintext highlighter-rouge">test</code> for release candidates (ie, <code class="language-plaintext highlighter-rouge">... stretch stable edge</code>).
Replace <code class="language-plaintext highlighter-rouge">stretch</code> with <code class="language-plaintext highlighter-rouge">jessie</code> for Jessie installs.</p>
<p>(<strong>Update 2017-09-29:</strong> If you’re installing EE, replace <code class="language-plaintext highlighter-rouge">https://download.docker.com/linux/debian</code> with your <code class="language-plaintext highlighter-rouge"><DOCKER-EE-SUBSCRIPTION-URL>/ubuntu</code> and use an Ubuntu suite like <code class="language-plaintext highlighter-rouge">xenial</code> which matches your host.)</p>
<p>At this point, you should be safe to run <code class="language-plaintext highlighter-rouge">apt-get update</code> to verify the changes:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get update
<span class="c">...
</span><span class="go">Get:5 https://download.docker.com/linux/debian stretch/stable amd64 Packages [1227 B]
</span><span class="c">...
</span><span class="go">Reading package lists... Done
</span></code></pre></div></div>
<p>(There shouldn’t be any warnings or errors about missing keys, etc.)</p>
<h1 id="configure-docker">configure Docker</h1>
<blockquote>
<p>This step could be done after Docker’s installed (and indeed, that’s usually when I do it because I forget that I should until I’ve got Docker installed and realize that my configuration is suboptimal), but doing it before ensures that Docker doesn’t have to be restarted later.</p>
</blockquote>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /etc/docker
<span class="nb">sudo </span>sensible-editor /etc/docker/daemon.json
</code></pre></div></div>
<blockquote>
<p>(<code class="language-plaintext highlighter-rouge">sensible-editor</code> can be replaced by whatever editor you prefer, but that command should choose or prompt for a reasonable default)</p>
</blockquote>
<blockquote>
<p>I then fill <code class="language-plaintext highlighter-rouge">daemon.json</code> with at least a default <code class="language-plaintext highlighter-rouge">storage-driver</code>. Whether I use <code class="language-plaintext highlighter-rouge">aufs</code> or <code class="language-plaintext highlighter-rouge">overlay2</code> depends on my kernel version and available modules – if I’m on Ubuntu, AUFS is still a no-brainer (since it’s included in the default kernel if the <code class="language-plaintext highlighter-rouge">linux-image-extra-XXX</code>/<code class="language-plaintext highlighter-rouge">linux-image-extra-virtual</code> package is installed), but on Debian AUFS is only available in either 3.x kernels (<code class="language-plaintext highlighter-rouge">jessie</code>’s default non-backports kernel) or recently in the <code class="language-plaintext highlighter-rouge">aufs-dkms</code> package (as of this writing, still only available on <code class="language-plaintext highlighter-rouge">stretch</code> and <code class="language-plaintext highlighter-rouge">sid</code> – no <code class="language-plaintext highlighter-rouge">jessie-backports</code> option).</p>
</blockquote>
<blockquote>
<p>If my kernel is 4.x+, I’m likely going to choose <code class="language-plaintext highlighter-rouge">overlay2</code> (or if that errors out, the older <code class="language-plaintext highlighter-rouge">overlay</code> driver).</p>
</blockquote>
<blockquote>
<p>Choosing an appropriate storage driver is a fairly complex topic, and I’d recommend that for serious production deployments, more research on pros and cons is performed than I’m including here (especially since AUFS and OverlayFS are <em>not</em> the only options – they’re just the two I personally use most often).</p>
</blockquote>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"storage-driver"</span><span class="p">:</span><span class="w"> </span><span class="s2">"overlay2"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h1 id="configure-boot-parameters">configure boot parameters</h1>
<blockquote>
<p>I usually set a few boot parameters as well (in <code class="language-plaintext highlighter-rouge">/etc/default/grub</code>’s <code class="language-plaintext highlighter-rouge">GRUB_CMDLINE_LINUX_DEFAULT</code> option – run <code class="language-plaintext highlighter-rouge">sudo update-grub</code> after adding these, space-separated).</p>
</blockquote>
<ul>
<li><code class="language-plaintext highlighter-rouge">cgroup_enable=memory</code> – enable “memory accounting” for containers (allows <code class="language-plaintext highlighter-rouge">docker run --memory</code> for setting hard memory limits on containers)</li>
<li><code class="language-plaintext highlighter-rouge">swapaccount=1</code> – enable “swap accounting” for containers (allows <code class="language-plaintext highlighter-rouge">docker run --memory-swap</code> for setting hard swap memory limits on containers)</li>
<li><code class="language-plaintext highlighter-rouge">systemd.legacy_systemd_cgroup_controller=yes</code> – newer versions of systemd <em>may</em> disable the legacy cgroup interfaces Docker currently uses; this instructs systemd to keep those enabled (for more details, see <a href="https://github.com/systemd/systemd/pull/4628">systemd/systemd#4628</a>, <a href="https://github.com/opencontainers/runc/issues/1175">opencontainers/runc#1175</a>, <a href="https://github.com/docker/docker/issues/28109">docker/docker#28109</a>)</li>
<li><code class="language-plaintext highlighter-rouge">vsyscall=emulate</code> – allow older binaries to run (<code class="language-plaintext highlighter-rouge">debian:wheezy</code>, etc.; see <a href="https://github.com/docker/docker/issues/28705">docker/docker#28705</a>)</li>
</ul>
<p>All together:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
<span class="nv">GRUB_CMDLINE_LINUX_DEFAULT</span><span class="o">=</span><span class="s2">"cgroup_enable=memory swapaccount=1 systemd.legacy_systemd_cgroup_controller=yes vsyscall=emulate"</span>
...
</code></pre></div></div>
<h1 id="install-docker">install Docker!</h1>
<p>Finally, the time has come.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install</span> <span class="nt">-V</span> docker-ce
<span class="c">...
</span><span class="go"> docker-ce (17.03.1~ce-0~debian-stretch)
</span><span class="c">...
</span><span class="go">
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>docker version
<span class="go">Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:07:28 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:07:28 2017
OS/Arch: linux/amd64
Experimental: false
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>usermod <span class="nt">-aG</span> docker <span class="s2">"</span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-un</span><span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>
<p>(<strong>Update 2017-09-29:</strong> If you’re installing EE, the package changes to <code class="language-plaintext highlighter-rouge">docker-ee</code>.)</p>
https://tianon.github.io/post/2016/12/07/docker-setupMy Docker Install Process2016-12-07T00:00:00-07:002016-12-07T00:00:00-07:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>I’ve had several requests recently for information about how I personally set up a new machine for running Docker (especially since I don’t use the infamous <code class="language-plaintext highlighter-rouge">curl get.docker.com | sh</code>), so I figured I’d outline the steps I usually take.</p>
<p>For the purposes of simplicity, I’m going to assume Debian (specifically <code class="language-plaintext highlighter-rouge">stretch</code>, the upcoming Debian stable release), but these should generally be easily adjustable to <code class="language-plaintext highlighter-rouge">jessie</code> or Ubuntu.</p>
<p>These steps <em>should</em> be fairly similar to what’s found in <a href="https://docs.docker.com/engine/installation/linux/debian/">upstream’s “Install Docker on Debian” document</a>, but do differ slightly in a few minor ways.</p>
<h1 id="grab-dockers-apt-repo-gpg-key">grab Docker’s APT repo GPG key</h1>
<p>The way I do this is probably a bit unconventional, but the basic gist is something like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">GNUPGHOME</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">mktemp</span> <span class="nt">-d</span><span class="si">)</span><span class="s2">"</span>
gpg <span class="nt">--keyserver</span> keyserver.ubuntu.com <span class="nt">--recv-keys</span> 58118E89F3A912897C070ADBF76221572C52609D
gpg <span class="nt">--export</span> <span class="nt">--armor</span> 58118E89F3A912897C070ADBF76221572C52609D | <span class="nb">sudo tee</span> /etc/apt/trusted.gpg.d/docker.gpg.asc
<span class="c"># gpg --export 58118E89F3A912897C070ADBF76221572C52609D | sudo tee /etc/apt/trusted.gpg.d/docker.gpg > /dev/null</span>
<span class="nb">rm</span> <span class="nt">-rf</span> <span class="s2">"</span><span class="nv">$GNUPGHOME</span><span class="s2">"</span>
</code></pre></div></div>
<p>(On <code class="language-plaintext highlighter-rouge">jessie</code> or another release whose APT doesn’t support <code class="language-plaintext highlighter-rouge">.asc</code> files in <code class="language-plaintext highlighter-rouge">/etc/apt/trusted.gpg.d</code>, I’d drop <code class="language-plaintext highlighter-rouge">--armor</code> and the <code class="language-plaintext highlighter-rouge">.asc</code> and go with simply <code class="language-plaintext highlighter-rouge">/.../docker.gpg</code>.)</p>
<p>This creates me a new GnuPG directory to work with (so my personal <code class="language-plaintext highlighter-rouge">~/.gnupg</code> doesn’t get cluttered with this new key), downloads Docker’s signing key from the keyserver gossip network (verifying the fetched key via the full fingerprint I’ve provided), exports the key into APT’s keystore, then cleans up the leftovers.</p>
<p>For completeness, other popular ways to fetch this include:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-key adv <span class="nt">--keyserver</span> keyserver.ubuntu.com <span class="nt">--recv-keys</span> 58118E89F3A912897C070ADBF76221572C52609D
</code></pre></div></div>
<p>(worth noting that <code class="language-plaintext highlighter-rouge">man apt-key</code> discourages the use of <code class="language-plaintext highlighter-rouge">apt-key adv</code>)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget <span class="nt">-qO-</span> <span class="s1">'https://apt.dockerproject.org/gpg'</span> | <span class="nb">sudo </span>apt-key add -
</code></pre></div></div>
<p>(no verification of the downloaded key)</p>
<p>Here’s the relevant output of <code class="language-plaintext highlighter-rouge">apt-key list</code> on a machine where I’ve got this key added in the way I outlined above:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>apt-key list
<span class="c">...
</span><span class="go">
/etc/apt/trusted.gpg.d/docker.gpg.asc
-------------------------------------
pub rsa4096 2015-07-14 [SCEA]
5811 8E89 F3A9 1289 7C07 0ADB F762 2157 2C52 609D
</span><span class="gp">uid [ unknown] Docker Release Tool (releasedocker) <docker@docker.com></span><span class="w">
</span><span class="go">
</span><span class="c">...
</span></code></pre></div></div>
<h1 id="add-dockers-apt-source">add Docker’s APT source</h1>
<p>If you prefer to fetch sources via HTTPS, install <code class="language-plaintext highlighter-rouge">apt-transport-https</code>, but I’m personally fine with simply doing GPG verification of fetched packages, so I forgo that in favor of less packages installed. YMMV.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s1">'deb http://apt.dockerproject.org/repo debian-stretch main'</span> | <span class="nb">sudo tee</span> /etc/apt/sources.list.d/docker.list
</code></pre></div></div>
<p>Hopefully it’s obvious, but <code class="language-plaintext highlighter-rouge">debian-stretch</code> in that line should be replaced by <code class="language-plaintext highlighter-rouge">debian-jessie</code>, <code class="language-plaintext highlighter-rouge">ubuntu-xenial</code>, etc. as desired. It’s also worth pointing out that this will <em>not</em> include Docker’s release candidates. If you want those as well, add <code class="language-plaintext highlighter-rouge">testing</code> after <code class="language-plaintext highlighter-rouge">main</code>, ie <code class="language-plaintext highlighter-rouge">... debian-stretch main testing' | ...</code>.</p>
<p>At this point, you should be safe to run <code class="language-plaintext highlighter-rouge">apt-get update</code> to verify the changes:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get update
<span class="c">...
</span><span class="go">Hit:1 http://apt.dockerproject.org/repo debian-stretch InRelease
</span><span class="c">...
</span><span class="go">Reading package lists... Done
</span></code></pre></div></div>
<p>(There shouldn’t be any warnings or errors about missing keys, etc.)</p>
<h1 id="configure-docker">configure Docker</h1>
<p>This step could be done after Docker’s installed (and indeed, that’s usually when I do it because I forget that I should until I’ve got Docker installed and realize that my configuration is suboptimal), but doing it before ensures that Docker doesn’t have to be restarted later.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /etc/docker
<span class="nb">sudo </span>sensible-editor /etc/docker/daemon.json
</code></pre></div></div>
<p>(<code class="language-plaintext highlighter-rouge">sensible-editor</code> can be replaced by whatever editor you prefer, but that command should choose or prompt for a reasonable default)</p>
<p>I then fill <code class="language-plaintext highlighter-rouge">daemon.json</code> with at least a default <code class="language-plaintext highlighter-rouge">storage-driver</code>. Whether I use <code class="language-plaintext highlighter-rouge">aufs</code> or <code class="language-plaintext highlighter-rouge">overlay2</code> depends on my kernel version and available modules – if I’m on Ubuntu, AUFS is still a no-brainer (since it’s included in the default kernel if the <code class="language-plaintext highlighter-rouge">linux-image-extra-XXX</code>/<code class="language-plaintext highlighter-rouge">linux-image-extra-virtual</code> package is installed), but on Debian AUFS is only available in either 3.x kernels (<code class="language-plaintext highlighter-rouge">jessie</code>’s default non-backports kernel) or recently in the <code class="language-plaintext highlighter-rouge">aufs-dkms</code> package (as of this writing, still only available on <code class="language-plaintext highlighter-rouge">stretch</code> and <code class="language-plaintext highlighter-rouge">sid</code> – no <code class="language-plaintext highlighter-rouge">jessie-backports</code> option).</p>
<p>If my kernel is 4.x+, I’m likely going to choose <code class="language-plaintext highlighter-rouge">overlay2</code> (or if that errors out, the older <code class="language-plaintext highlighter-rouge">overlay</code> driver).</p>
<p>Choosing an appropriate storage driver is a fairly complex topic, and I’d recommend that for serious production deployments, more research on pros and cons is performed than I’m including here (especially since AUFS and OverlayFS are <em>not</em> the only options – they’re just the two I personally use most often).</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"storage-driver"</span><span class="p">:</span><span class="w"> </span><span class="s2">"overlay2"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h1 id="configure-boot-parameters">configure boot parameters</h1>
<p>I usually set a few boot parameters as well (in <code class="language-plaintext highlighter-rouge">/etc/default/grub</code>’s <code class="language-plaintext highlighter-rouge">GRUB_CMDLINE_LINUX_DEFAULT</code> option – run <code class="language-plaintext highlighter-rouge">sudo update-grub</code> after adding these, space-separated).</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">cgroup_enable=memory</code> – enable “memory accounting” for containers (allows <code class="language-plaintext highlighter-rouge">docker run --memory</code> for setting hard memory limits on containers)</li>
<li><code class="language-plaintext highlighter-rouge">swapaccount=1</code> – enable “swap accounting” for containers (allows <code class="language-plaintext highlighter-rouge">docker run --memory-swap</code> for setting hard swap memory limits on containers)</li>
<li><code class="language-plaintext highlighter-rouge">systemd.legacy_systemd_cgroup_controller=yes</code> – newer versions of systemd <em>may</em> disable the legacy cgroup interfaces Docker currently uses; this instructs systemd to keep those enabled (for more details, see <a href="https://github.com/systemd/systemd/pull/4628">systemd/systemd#4628</a>, <a href="https://github.com/opencontainers/runc/issues/1175">opencontainers/runc#1175</a>, <a href="https://github.com/docker/docker/issues/28109">docker/docker#28109</a>)</li>
<li><code class="language-plaintext highlighter-rouge">vsyscall=emulate</code> – allow older binaries to run (<code class="language-plaintext highlighter-rouge">debian:wheezy</code>, etc.; see <a href="https://github.com/docker/docker/issues/28705">docker/docker#28705</a>)</li>
</ul>
<p>All together:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
<span class="nv">GRUB_CMDLINE_LINUX_DEFAULT</span><span class="o">=</span><span class="s2">"cgroup_enable=memory swapaccount=1 systemd.legacy_systemd_cgroup_controller=yes vsyscall=emulate"</span>
...
</code></pre></div></div>
<h1 id="install-docker">install Docker!</h1>
<p>Finally, the time has come.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install</span> <span class="nt">-V</span> docker-engine
<span class="c">...
</span><span class="go">
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>docker version
<span class="go">Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 21:45:16 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 21:45:16 2016
OS/Arch: linux/amd64
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>usermod <span class="nt">-aG</span> docker <span class="s2">"</span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-un</span><span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>
<p>(Reboot or logout/login to update your session to include <code class="language-plaintext highlighter-rouge">docker</code> group membership and thus no longer require <code class="language-plaintext highlighter-rouge">sudo</code> for using <code class="language-plaintext highlighter-rouge">docker</code> commands.)</p>
<p>Hope this is useful to someone! If nothing else, it’ll serve as a concise single-page reference for future-tianon. 😇</p>
<ul>
<li><strong>Updated 2017-04-11</strong>: adjusted some commands to be easier to munge for other platforms (especially so I stop screwing up the <code class="language-plaintext highlighter-rouge">gpg --export</code> line and getting garbage to my terminal)</li>
</ul>
https://tianon.github.io/post/2016/03/03/vultr-docker-ipv6Docker on VULTR + IPv62016-03-03T00:00:00-07:002016-03-03T00:00:00-07:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>I’ve been using <a href="https://www.vultr.com">VULTR</a> for a little while now and have been generally very pleased (especially with the very recent facelift the management portal received). I don’t want to waste too much time talking about it, but the “killer feature” for me (over some of their competitors like <a href="https://www.digitalocean.com">DigitalOcean</a>) is that I can provide a raw ISO and provision my VM directly using it as I would any local VM (which also means that once my VM is up and working, I get to use the OS’s standard kernel, which is especially important for using Debian Unstable well).</p>
<p>Anyhow, already too much about that – let’s get to the cool stuff.</p>
<p>Getting right down to the beef, let’s assume I’ve got a VULTR instance already created, my OS is already installed and working, I’ve enabled IPv6 within VULTR, ensured that my VM is able to <code class="language-plaintext highlighter-rouge">ping6 google.com</code> (to verify at least basic routability), <em>and</em> have Docker version 1.10.2 installed.</p>
<p>For the sake of demonstration, we’ll assume that VULTR has assigned my IPv6 as follows: (available under the VM details via <code class="language-plaintext highlighter-rouge">Settings > IPv6</code>)</p>
<ul>
<li>Default IP: <code class="language-plaintext highlighter-rouge">2001:db8::5400:00ff:fe20:2295</code></li>
<li>Network: <code class="language-plaintext highlighter-rouge">2001:db8::</code></li>
<li>CIDR: 64</li>
</ul>
<p>(The astute reader may recognize <a href="https://tools.ietf.org/html/rfc3849">RFC3849</a> here. 😏)</p>
<p>The relevant documentation which helped me get to the working state outlined below is in <a href="https://docs.docker.com/engine/userguide/networking/default_network/ipv6/">the “IPv6 with Docker” section</a>.</p>
<p>The first step I took was creating a systemd drop-in file so that I could modify the daemon startup parameters (to include <code class="language-plaintext highlighter-rouge">--ipv6</code> and <code class="language-plaintext highlighter-rouge">--fixed-cidr-v6</code>):</p>
<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># /etc/systemd/system/docker.service.d/ipv6.conf
</span><span class="nn">[Service]</span>
<span class="py">ExecStart</span><span class="p">=</span>
<span class="py">ExecStart</span><span class="p">=</span><span class="s">/usr/bin/docker daemon -H fd:// --ipv6 --fixed-cidr-v6 2001:db8::/80</span>
</code></pre></div></div>
<p>I chose to use just <code class="language-plaintext highlighter-rouge">/80</code> for Docker – any other reasonable prefix (assuming it is routed to your host / host network) should also work; the documentation I linked above has an example using a <code class="language-plaintext highlighter-rouge">/125</code>, for example.</p>
<p>With this half in place, I can <code class="language-plaintext highlighter-rouge">systemctl daemon-reload</code> and <code class="language-plaintext highlighter-rouge">systemctl restart docker.service</code>, and when I start a container it will be automatically assigned an IPv6 address from within that prefix. Excellent.</p>
<p>An important caveat to note is that this <em>will</em> break discovery on our host due to Docker enabling forwarding for us, so (assuming your “internet-facing” interface is named <code class="language-plaintext highlighter-rouge">ens3</code> for the sake of illustration; it might just as easily be <code class="language-plaintext highlighter-rouge">eth0</code>, <code class="language-plaintext highlighter-rouge">eth1</code>, <code class="language-plaintext highlighter-rouge">enps3</code>, <code class="language-plaintext highlighter-rouge">lan0</code>, <code class="language-plaintext highlighter-rouge">wlan0</code>, etc) I had to <code class="language-plaintext highlighter-rouge">sysctl net.ipv6.conf.ens3.accept_ra=2</code>, and I added it to <code class="language-plaintext highlighter-rouge">/etc/sysctl.d/docker-ipv6.conf</code> for good measure (so that I don’t lose it after I reboot).</p>
<p>The second half of our IPv6 to containers problem is routing. The nitty-gritty details of this are discussed in <a href="https://docs.docker.com/engine/userguide/networking/default_network/ipv6/#using-ndp-proxying">the “Using NDP proxying” section</a> of the documentation, but the gist is that my containers have IPv6 addresses, but the outside world doesn’t have a route that leads to them, and that we need to tell the kernel to respond to solicitations for our container’s IPv6 addresses appropriately.</p>
<p>The kernel has a mechanism for doing so (via <code class="language-plaintext highlighter-rouge">ip -6 neigh ...</code>), but it is limited to individual addresses and is thus not especially great for having a solution that works “magically” without further manual labor per-container.</p>
<p>This is where <a href="https://github.com/DanielAdolfsson/ndppd">ndppd</a> (also <a href="https://packages.debian.org/sid/ndppd">packaged for Debian as <code class="language-plaintext highlighter-rouge">ndppd</code></a>) came in.</p>
<div class="language-nginx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># /etc/ndppd.conf</span>
<span class="k">proxy</span> <span class="s">ens3</span> <span class="p">{</span>
<span class="kn">rule</span> <span class="mi">2001</span><span class="p">:</span><span class="s">db8::/80</span> <span class="p">{</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>After getting this configuration in place and restarting <code class="language-plaintext highlighter-rouge">ndppd</code> (<code class="language-plaintext highlighter-rouge">systemctl restart ndppd</code>), magic happened. My containers could <code class="language-plaintext highlighter-rouge">ping6 google.com</code>, and my other IPv6 hosts could <code class="language-plaintext highlighter-rouge">ping6</code> the IPv6 addresses of my individual containers!</p>
<p>You’ve probably noted that this configuration isn’t exactly secure, since it means that each of my individual containers has a <em>publicly</em> routable IPv6 address, but for this specific use case, I’m OK with that! 🍦</p>
<p><strong>Update</strong> (2015-03-09): thanks to Алексей Шилин for correcting my systemd drop-in file usage! ♥</p>
https://tianon.github.io/post/2015/05/28/dep8-tldrDEP8 - TL;DR2015-05-28T00:00:00-06:002015-05-28T00:00:00-06:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>DEP stands for “Debian Enhancement Proposals”. <a href="http://dep.debian.net/deps/dep8/">DEP8</a> is about package testing, specifically post-install (as opposed to <code class="language-plaintext highlighter-rouge">dh_auto_test</code> which runs during package build, usually for unit tests). It’s great for integration tests, etc. that have more interesting requirements for running than unit tests normally do.</p>
<p>The problem is that <a href="http://anonscm.debian.org/gitweb/?p=autopkgtest/autopkgtest.git;a=blob_plain;f=doc/README.package-tests.rst;hb=HEAD">the spec</a> is a little bit long in the tooth for casual reading / understanding-at-a-glance.</p>
<p>What follows is my own personal TL;DR version.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">cd </span>your-package/
<span class="gp">$</span><span class="w"> </span><span class="nb">mkdir</span> <span class="nt">-p</span> debian/tests
<span class="gp">$</span><span class="w"> </span>vim debian/tests/control
</code></pre></div></div>
<p>(editor of your choice)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Tests: my-test
Depends: hello, @
Restrictions: needs-root
</code></pre></div></div>
<p>(see <a href="https://anonscm.debian.org/gitweb/?p=autopkgtest/autopkgtest.git;a=blob_plain;f=doc/README.package-tests.rst;hb=HEAD">the spec</a> for more info about what these mean and what valid values are)</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">touch </span>debian/tests/my-test
<span class="gp">$</span><span class="w"> </span><span class="nb">chmod</span> +x debian/tests/my-test
<span class="gp">$</span><span class="w"> </span>vim debian/tests/my-test
</code></pre></div></div>
<p>(editor of your choice)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="nb">set</span> <span class="nt">-e</span>
hello
<span class="c"># other bits testing your actual package (installed because of "@" in "Depends:")</span>
</code></pre></div></div>
<p>(again, see the spec linked above for details of how this script should behave – in general, non-zero exit code or stderr output mean failure)</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>apt-get <span class="nb">install </span>autopkgtest <span class="c"># if not already installed</span>
<span class="gp">$</span><span class="w"> </span>adt-run <span class="nt">--unbuilt-tree</span> <span class="nb">.</span> <span class="nt">---</span> VIRT-SERVER
</code></pre></div></div>
<p>Where <code class="language-plaintext highlighter-rouge">VIRT-SERVER</code> is one of: (as of this writing and installed by default with <code class="language-plaintext highlighter-rouge">autopkgtest</code> – YMMV)</p>
<ul>
<li><a href="http://manpages.debian.org/cgi-bin/man.cgi?manpath=Debian+unstable+sid&query=adt-virt-chroot"><code class="language-plaintext highlighter-rouge">chroot</code></a></li>
<li><a href="http://manpages.debian.org/cgi-bin/man.cgi?manpath=Debian+unstable+sid&query=adt-virt-lxc"><code class="language-plaintext highlighter-rouge">lxc</code></a></li>
<li><a href="http://manpages.debian.org/cgi-bin/man.cgi?manpath=Debian+unstable+sid&query=adt-virt-null"><code class="language-plaintext highlighter-rouge">null</code></a></li>
<li><a href="http://manpages.debian.org/cgi-bin/man.cgi?manpath=Debian+unstable+sid&query=adt-virt-qemu"><code class="language-plaintext highlighter-rouge">qemu</code></a></li>
<li><a href="http://manpages.debian.org/cgi-bin/man.cgi?manpath=Debian+unstable+sid&query=adt-virt-schroot"><code class="language-plaintext highlighter-rouge">schroot</code></a></li>
<li><a href="http://manpages.debian.org/cgi-bin/man.cgi?manpath=Debian+unstable+sid&query=adt-virt-ssh"><code class="language-plaintext highlighter-rouge">ssh</code></a></li>
</ul>
https://tianon.github.io/post/2015/05/25/dns.he.net-dd-wrtDD-WRT + dns.he.net (DDNS / inadyn)2015-05-25T00:00:00-06:002015-05-25T00:00:00-06:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>The DD-WRT wiki hilariously has <a href="https://www.dd-wrt.com/wiki/index.php/Dynamic_DNS#he.net">a page about this</a>, but it’s not very helpful and <a href="https://www.dd-wrt.com/phpBB2/viewtopic.php?t=137570">account creation is entirely disabled</a>, so here goes a blog post for my own future reference:</p>
<p>In the <a href="https://dns.he.net">dns.he.net control panel</a>, enable the hostname (<code class="language-plaintext highlighter-rouge">HOSTNAME</code>) for “dynamic dns”. Click the DDNS icon and generate a “key” (<code class="language-plaintext highlighter-rouge">KEY</code>), which will be used as the password for updating.</p>
<p>In the DD-WRT control panel, under “Setup > DDNS” (at least in <code class="language-plaintext highlighter-rouge">Firmware: DD-WRT v24-sp2 (02/19/14) std</code>):</p>
<ul>
<li>DDNS Service: <code class="language-plaintext highlighter-rouge">Custom</code></li>
<li>DYNDNS Server: <code class="language-plaintext highlighter-rouge">dyn.dns.he.net</code></li>
<li>Username: <code class="language-plaintext highlighter-rouge">HOSTNAME</code></li>
<li>Password: <code class="language-plaintext highlighter-rouge">KEY</code></li>
<li>Hostname: <code class="language-plaintext highlighter-rouge">HOSTNAME</code></li>
<li>URL: <code class="language-plaintext highlighter-rouge">/nic/update?hostname=</code></li>
</ul>
<p>Whala.</p>
<p><strong>Update</strong> (2015-08-19):</p>
<p><code class="language-plaintext highlighter-rouge">ddclient.conf</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>protocol=dyndns2
use=if
if=eth0
server=dyn.dns.he.net
ssl=no
login=HOSTNAME
password=KEY
HOSTNAME
</code></pre></div></div>
https://tianon.github.io/post/2014/11/22/internets-own-boyThe Internet's Own Boy2014-11-22T00:00:00-07:002014-11-22T00:00:00-07:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>I am really late coming to this train. This has been on my list since the
day it was released, and today I finally found the time to sit alone and
digest.</p>
<p>To say that it was “emotionally moving” would be a gross misrepresentation
of the film. There were clearly some aspects added for dramatic effect, but
if you strip away (for example) the demonization of the opposition’s
actions, the proceedings remain absolutely <em>astounding</em> and the results
entirely heartbreaking.</p>
<p>That a society which claims to be as advanced as ours does (technologically,
morally, socially, militarily, etc) could demoralize an individual in this
manner seems terminally and criminally corrupt.</p>
<p>When my filesystem gets corrupted, I try to desperately salvage any of the
useful information (usually while trying to salvage the entire filesystem),
and once my efforts have proven to have provided all the fruits they
possibly can, I proceed to format the drive and begin again. Sometimes, a
system needs to be rebooted. A fresh install of the operating system
usually extends the life of a system by a measurable amount of time.</p>
<p>Perhaps reinstalling our base operating system is worth a try?</p>
https://tianon.github.io/post/2014/08/30/debconf14DebConf142014-08-30T00:00:00-06:002014-08-30T00:00:00-06:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>I was given the opportunity to attend DebConf in Portland this year, and I
must say that it took me entirely by surprise (as a first-time attendee).</p>
<p>Most conferences have several strong talk tracks and you end up spending a
lot of time sitting in talks wondering when they’ll be over. At DebConf,
there’s an entirely different dynamic, with a strong focus on what they like
to call “the hallway track” (which this year has taken place a lot in the
hacklabs, too). Everyone here wants to either talk about building cool
stuff, or sit down and actually build some cool stuff. A large number of
the talks are just launchpads for informal Q+As or actual hack sessions.</p>
<p>By coming, I’ve learned a lot about the Debian community and how it operates
as a whole, and managed to meet a lot of very interesting and cool people
(not to mention getting a bunch of them to sign my GPG key, which is also
nice).</p>
<p>Hopefully I’ll be able to attend more DebConfs in the future, because I’ve
had a great time!</p>
https://tianon.github.io/post/2014/05/17/docker-on-gentooLove is a Battlefield (Docker on Gentoo)2014-05-17T00:00:00-06:002014-05-17T00:00:00-06:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>Docker on Gentoo can be a beautiful thing, but it can also be a challenge
navigating some of the trade-offs.</p>
<p>The hardest decision to make, in my opinion, is which storage backend to
use. Each one has ups and downs, and some of them have ups and downs that
are more specific to Gentoo than others.</p>
<h1 id="aufs">“aufs”</h1>
<p>Normally for an out-of-kernel module (even a filesystem), it would be a
simple matter to simply compile said module against the proper kernel
sources and load it up; no harm, no foul. What’s particularly needling
about AUFS is that it requires patches to the kernel proper (which, I might
add, were submitted for inclusion in the kernel and rejected).</p>
<p>The quandary that’s most interesting about AUFS is that it’s currently the
recommended Docker backend. For Ubuntu and Debian users, this isn’t a
problem since the AUFS patches are included in the main kernels and so the
<code class="language-plaintext highlighter-rouge">aufs</code> module is merely a single <code class="language-plaintext highlighter-rouge">apt-get install</code> away.</p>
<p>As you might imagine, these patches make a bit of a stir for someone who
builds their own kernels (like, say, a Gentoo user), and there are two main
ways to get them.</p>
<h2 id="sys-kernelaufs-sources"><code class="language-plaintext highlighter-rouge">sys-kernel/aufs-sources</code></h2>
<p>I’ll start with the easy way. If you <code class="language-plaintext highlighter-rouge">emerge sys-kernel/aufs-sources</code>,
you’ll get <code class="language-plaintext highlighter-rouge">sys-kernel/gentoo-sources</code> with the AUFS patches pre-applied.
Choosing this method, it’s merely a matter of making sure <code class="language-plaintext highlighter-rouge">CONFIG_AUFS_FS</code>
is enabled in your <code class="language-plaintext highlighter-rouge">.config</code> and you’re good to go. If you’re already using
stock <code class="language-plaintext highlighter-rouge">sys-kernel/gentoo-sources</code> and/or are not averse to a slight change,
this will be the easiest, cleanest, and most importantly the least
error-prone option by far.</p>
<h2 id="sys-fsaufs3"><code class="language-plaintext highlighter-rouge">sys-fs/aufs3</code></h2>
<p>The alternative is to use <code class="language-plaintext highlighter-rouge">sys-fs/aufs3</code>. This package provides both the
necessary kernel patches and compiles the <code class="language-plaintext highlighter-rouge">aufs</code> module, making it much more
suitable to <code class="language-plaintext highlighter-rouge">sys-kernel/vanilla-sources</code> and the like. The <code class="language-plaintext highlighter-rouge">aufs</code> module
will only load on a kernel compiled with the AUFS patches. This ebuild
includes a <code class="language-plaintext highlighter-rouge">kernel-patch</code> use flag that will automatically apply the patches
to <code class="language-plaintext highlighter-rouge">/usr/src/linux</code> at merge time, which is the simplest way to ensure they
are applied.</p>
<p>Note that in my experience, this method is very human error-prone. Using
<code class="language-plaintext highlighter-rouge">sys-kernel/aufs-sources</code>, portage tracks the patches. Using
<code class="language-plaintext highlighter-rouge">sys-fs/aufs3</code>, it’s all up to you. I wish I could get back the lost time
rebooting into a new kernel only to realize I hadn’t recompiled it again
after re-emerging <code class="language-plaintext highlighter-rouge">sys-fs/aufs3</code>.</p>
<h1 id="btrfs">“btrfs”</h1>
<p>BTRFS is fun. It’s speedy, it’s hip, it’s experimental. The obvious
downside to using it as your Docker backend is that most of us don’t have
our root filesystem on it, which means we either have to reinstall our OS,
make a new partition/drive/loopback for Docker, or choose a different
backend.</p>
<p>Note that if you <em>do</em> have BTRFS as your root filesystem, you want to make
sure you <em>do not</em> use the AUFS backend. AUFS on top of BTRFS has lots and
lots of strange issues.</p>
<h1 id="devicemapper">“devicemapper”</h1>
<p>The LVM/devicemapper backend is especially cool because the kernel features
it requires are enabled in a wide variety of pre-compiled kernels, making
this by far the easiest backend to get started with. Also, it doesn’t
play foul with any known filesystems since it effectively mounts containers
in loopback, avoiding potential issues with filesystems interfering.</p>
<p>However, unless you configure it to use a raw physical disk partition, the
performance will likely leave much to be desired.</p>
<h1 id="vfs">“vfs”</h1>
<p>What we lovingly refer to as “vfs” is an interesting driver. It’s what’s
used for volumes, and is essentially a reference implementation for graph
drivers. It has no “copy on write” at all, and is essentially just “copy
the entire rootfs for each new layer”, so is perfectly suited for volumes,
but is not at all well-suited for being the general daemon backend.</p>
https://tianon.github.io/post/2014/04/19/getting-high-on-hyGetting High on Hy2014-04-19T00:00:00-06:002014-04-19T00:00:00-06:00Tianon Graviadmwiggin@gmail.comhttps://tianon.github.io<p>My good friend <a href="http://pault.ag">Paul Tagliamonte</a> recently gave a talk at PyCon
2014 about his language, Hy. It’s effectively Lisp implemented inside Python
(so cleanly that Python itself doesn’t care to differentiate the result from any
other Python code).</p>
<p>It’s a solid talk that covers a lot of good ground about some of the cool stuff
you can do with Hy, and especially about the internals of exactly how Hy works,
which is really fascinating stuff. There’s even a shout-out to our shared love,
<a href="https://www.docker.io">Docker</a>!</p>
<p>The video can be found on
<a href="https://www.youtube.com/watch?v=AmMaN1AokTI">YouTube</a>, but I’ve also embedded
it below for your viewing pleasure.</p>
<iframe width="100%" height="400" src="//www.youtube.com/embed/AmMaN1AokTI?start=115&html5=1&rel=0" frameborder="0" allowfullscreen=""></iframe>
<p>If you’d like to give Hy a try, you can check it out with
<a href="http://try-hy.appspot.com">try-hy</a>, which is Hy running sandboxed on Google App
Engine so you can play with it freely inside your browser.</p>
<p><a href="https://github.com/hylang/hy/blob/master/eg/sh/tagwords.hy">sh/tagwords.hy</a>:</p>
<div class="language-lisp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">;; python-sh from hy</span>
<span class="p">(</span><span class="nb">import</span> <span class="nv">[sh</span> <span class="nv">[cat</span> <span class="nv">grep]]</span><span class="p">)</span>
<span class="p">(</span><span class="nb">print</span> <span class="s">"Words that end with `tag`:"</span><span class="p">)</span>
<span class="p">(</span><span class="nb">print</span> <span class="p">(</span><span class="nv">-></span> <span class="p">(</span><span class="nv">cat</span> <span class="s">"/usr/share/dict/words"</span><span class="p">)</span> <span class="p">(</span><span class="nv">grep</span> <span class="s">"-E"</span> <span class="s">"tag$"</span><span class="p">)))</span>
</code></pre></div></div>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>hy sh/tagwords.hy
<span class="go">Words that end with `tag`:
Bundestag
Maytag
Reichstag
Sontag
ragtag
stag
tag
</span></code></pre></div></div>