Removing IME: Upgrading System76 firmware on Arch

2018-04-11T00:00:00-07:00

In the wake of the Intel Management Engine security revelations, System76 developed a firmware upgrade for all of its machines and released an update plan in November 2017. In February 2018, owners of the Oryx Pro were informed that the firmware update was available through System76’s open source firmware updater. For anyone not on System76’s Pop!_OS or similar Debian-based distros, this firmware updater probably didn’t do anything. After waiting patiently for a couple of months for more updates and not seeing any fixes, I dug into how I could get things going. Herein lies the easiest way I found.

Before proceeding

To start with, try the firmware update normally, as recommended by the System76 docs. The approach I took is pretty hacky, but it was the only thing which worked for my setup. Ideally, you can take a more trodden route.

Silent errors when running

The system76-driver AUR package provides a systemd service which just runs the system76-firmware command to check for firmware updates. In my case, when I ran it, I saw something like the following:

$ system76-firmware
2018-04-08 10:48:51,895  INFO  Verified manifest signature...
2018-04-08 10:48:51,895  INFO  Fetching f7cd3816401c6ab1cd2f0a83285a56ee432a9736b707870fb7aeb34c2750bcefc2adcf0f83952696eb688b9768e93f68 with cache /var/cache/system76-firmware
2018-04-08 10:48:51,897  INFO  Fetching e4206477b3f5bad09d54363a78ae79e2916127399e9725c3b9d77bf229c25c293111926d841c1c05b186a96c0963f6ff with cache /var/cache/system76-firmware
2018-04-08 10:48:51,933  INFO  Fetching ec0b0b475412acde6b2b9a05647a64f48beaa5baea298e8801ce1a34bbddcdada5fc2d8025b2e6f07d9802c384a01e7c with cache /var/cache/system76-firmware
2018-04-08 10:48:52,262  INFO  Verified manifest signature...
2018-04-08 10:48:52,263  INFO  Fetching f7cd3816401c6ab1cd2f0a83285a56ee432a9736b707870fb7aeb34c2750bcefc2adcf0f83952696eb688b9768e93f68 with cache /var/cache/system76-firmware
2018-04-08 10:48:52,265  INFO  Fetching e4206477b3f5bad09d54363a78ae79e2916127399e9725c3b9d77bf229c25c293111926d841c1c05b186a96c0963f6ff with cache /var/cache/system76-firmware
2018-04-08 10:48:52,299  INFO  Fetching ec0b0b475412acde6b2b9a05647a64f48beaa5baea298e8801ce1a34bbddcdada5fc2d8025b2e6f07d9802c384a01e7c with cache /var/cache/system76-firmware
$ echo $?
0

No errors, no warnings, a clean exit code, but it didn’t update my firmware or give me any insight into how or when that was going to happen. This is how things were since the system76-firmware command was added to the AUR package.

Issues with the display

When deciding to dig into what was going on, the first thing I did was step through the program with pdb. From there, it became clear that system76-firmware is trying to show a modal, but can’t find the display name, so it exits silently. It’s reading the display name from who, which is odd, since I’d think it’d just check DISPLAY, so I edited the source to just return ":0", which is the value of echo $DISPLAY in my X session. This could be because I’m running i3-wm and not GNOME or KDE, but it’s an assumption made on System76’s part either way.

For the file: /usr/lib/python3.6/site-packages/system76driver/firmware.py

diff --git a/firmware.py b/firmware.py
index 95bafe1..90c60f9 100644
--- a/firmware.py
+++ b/firmware.py
@@ -444,6 +444,7 @@ def get_user_session():
                     "who | awk -v vt=tty$(fgconsole) '$0 ~ vt {print $5}'",
                     shell=True
                 ).decode('utf-8').rstrip('\n').lstrip('(').rstrip(')')
+    display_name = ":0" # XXX: Hack
 
     user_pid = subprocess.check_output(
                     "who -u | awk -v vt=tty$(fgconsole) '$0 ~ vt {print $6}'",

Issues with efibootmgr

After that one line fix, running system76-firmware resulted in a modal! I could choose to close the modal or click a button to install the firmware. After trying to install, another modal shows up with the procedure instructions, as they’re also shown on the System76 support page, and a button to restart into the firmware updater. Restart!

…

Back into GRUB, then into Arch, with no firmware updates. Damn. So I dug more into the source for, hopefully, another one liner. There’s a bash script, as part of the Python source, called FIRMWARE_SET_NEXT_BOOT. It looks like this:

EFIDEV="$(findmnt -n /boot/efi -o SOURCE)"
EFINAME="$(basename "${EFIDEV}")"
EFISYS="$(readlink -f "/sys/class/block/${EFINAME}")"
EFIPART="$(cat "${EFISYS}/partition")"
DISKSYS="$(dirname "${EFISYS}")"
DISKNAME="$(basename "${DISKSYS}")"
DISKDEV="/dev/${DISKNAME}"

echo -e "\e[1mCreating Boot1776 on "${DISKDEV}" "${EFIPART}" \e[0m" >&2
efibootmgr -B -b 1776 || true
efibootmgr -C -b 1776 -d "${DISKDEV}" -p "${EFIPART}" -l '\\system76-firmware-update\\boot.efi' -L "system76-firmware-update"

echo -e "\e[1mSetting BootNext to 1776\e[0m" >&2
efibootmgr -n 1776

echo -e "\e[1mInstalled system76-firmware-update\e[0m" >&2
efibootmgr -v

This is what’s trying to make the next boot go into the updater’s EFI file instead of the normal boot order. So, already, there are some assumptions made.

The user is booting with EFI
/boot/efi is mounted
Writing the System76 updater EFI to /boot/efi/system76-firmware-update/ is useful
efibootmgr -C is valid
Errors don’t matter. They do though, so it should be using safer settings.

For all but the first of these assumptions, on my Arch setup, System76 is guessing incorrectly. Yes, I’m booting with EFI, but /boot/efi isn’t always mounted. Furthermore, the system76-firmware-update directory should be installed into /boot/efi/EFI if it wants to actually be useful. Finally, according to the efibootmgr man page, -c should be used to create boot entries. -C doesn’t exist. In your system76-firmware output, you’ll probably see something like this (from efibootmgr):

Could not prepare Boot variable: No such file or directory

So, you might try the following, in order to get your restarts to bring you into the updater:

Patch the bash script to use -c instead of -C with efibootmgr
Mount /boot/efi and then run system76-firmware (then see if it installs properly and try rebooting)
Move /boot/efi/system76-firmware-update to /boot/efi/EFI/system76-firmware-update and then run system76-firmware again, to see if efibootmgr -c picks it up and try your restart

I was able to get efibootmgr to finally run without error, but a restart still just brought me to GRUB. So, I took a more manual route.

Before proceeding: get Rescatux

Very importantly, make sure you have a USB or CD/DVD with a burned image of Rescatux. Rescatux is an amazing 20MB rescue boot image which can boot into most anything. This is not a precaution; if you follow these steps, you will need Rescatux, so burn it now.

Download it here
Burn with dd or whatever you prefer

Manually loading the updater EFI

My idea was simple: just add a GRUB entry for the firmware updater, since I can so reliably make it into GRUB. The catch is that the firmware updater, thinking that it was booted from the temporary boot path made by the bash script discussed above, deletes the EFI boot path which was used to boot. As long as there’s a Rescatux image lying around, though, that’s not a problem.

Add the GRUB entry

The GRUB entry should look something like this (put it at the bottom of /etc/grub.d/40_custom):

menuentry "System76 Firmware Update" {
        insmod chain
        insmod search_fs_uuid
        search --fs-uuid --no-floppy --set=root TODO-ID
        chainloader /EFI/system76-firmware-update/boot.efi
}

In order for it to work for your machine, you need to do two things.

Find the filesystem UUID

Run cfdisk <your disk> and select your EFI partition to see its filesystem id. For example, mine is 129D-B845. Replace TODO-ID in the above GRUB entry with your id. Here’s what it should look like:

$ cfdisk /dev/nvme0n1
                                   Disk: /dev/nvme0n1
                 Size: 232.9 GiB, 250059350016 bytes, 488397168 sectors
              Label: gpt, identifier: B7545E19-CF68-4779-BD95-F12C7DC8DDA6

    Device                 Start          End      Sectors      Size Type
>>  /dev/nvme0n1p1          2048      1050623      1048576      512M EFI System         
    /dev/nvme0n1p2       1050624    488397134    487346511    232.4G Linux filesystem
 ┌────────────────────────────────────────────────────────────────────────────────────┐
 │  Partition name: primary                                                           │
 │  Partition UUID: 7761E4D0-5726-45BE-9761-25FC114DE691                              │
 │  Partition type: EFI System (C12A7328-F81F-11D2-BA4B-00A0C93EC93B)                 │
 │ Filesystem UUID: 129D-B845                                                         │
 │Filesystem LABEL: EFI                                                               │
 └────────────────────────────────────────────────────────────────────────────────────┘
   [ Delete ]  [ Resize ]  [  Quit  ]  [  Type  ]  [  Help  ]  [  Write ]  [  Dump  ]

Move the updater to the EFI partition

As mentioned above, system76-firmware installs the updater to /boot/efi/system76-firmware-update. If that’s your EFI root, great. Just update the GRUB entry to not use /EFI/. If your system is like mine, though, you’ll need to move the updater to /boot/efi/EFI/system76-firmware-update.

Generate GRUB’s new config

With the new GRUB entry, the final step to get ready for the reboot is to generate the new GRUB configs.

$ grub-mkconfig --output /boot/grub/grub.cfg

Time to update the firmware

At this point, a reboot should bring you into GRUB with your new entry. After selecting your new entry, you should see the updater and you can press enter to continue. Remember to have the System76 docs on hand, so you understand how the update flow should go.

After a little while, the updater will reboot. Your fans will be on full blast. At the GRUB menu, choose the updater again. Once you’re back in the updater, it’ll work for a while more, run some checks, and finally power off. Follow the System76 docs to jump into BIOS, adopt the defaults, and everything else. When you’re ready to get back into GNU/Linux, you’ll find your EFI boot path is gone.

Rescatux

Plug in your Rescatux media, reboot your machine, optionally hit F2 to change your boot order, if necessary, to make it into the Rescatux menu. The default option, to find all bootable OSs, will do the trick, so just hit enter and then choose the first Linux boot option to get back into your typical environment.

Reinstall GRUB

Since your EFI boot path was wiped by the updater and you probably don’t want to boot via Rescatux forever, once you’re back in a sane GNU/Linux environment, just mount your EFI partition and reinstall GRUB. For me, that process looked like this.

$ cfdisk /dev/nvme0n1
$ mount /dev/nvme0n1p1 /boot/efi
$ grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=grub

The Intel Management Engine has now been disabled and its code has been removed. You can try running the firmware updater again, just to be sure that it detects everything is up to date.

Cleanup

Finally, feel free to remove /boot/efi/EFI/system76-firmware-update and /boot/efi/system76-firmware-update, if they’re still around. You can also remove the GRUB boot entry from /etc/grub.d/40_custom and re-generate your GRUB config.

$ grub-mkconfig --output /boot/grub/grub.cfg

Post-update issues

In #system76 on Freenode, which is barren and likely not worth joining, I’ve seen some questions about people losing some fn key functionality after the update. For me, HDMI didn’t work at all after the update. I unplugged and tested the HDMI cable and monitor on another machine and concluded that the firmware update must’ve borked something. xrandr didn’t show any connections, but I didn’t look further into it than that, since it was late and I was pleased enough to’ve finished the update.

In the morning, HDMI worked as expected, without any hiccups. Maybe the machine also just need a good night’s rest.

Switching from solarized to gruvbox

2018-02-01T00:00:00-08:00

Ever since 2012, I’ve been using solarized dark for everything I can possibly configure. From my web browser to my terminal, and everything in between, I was a solarized user. Alas, I was experiencing regular eye strain, especially at night, even using solarized dark with low monitor brightness. I use a large font size (large enough for people to comment on it regularly) and, as of my recent tests, I have at least 20/20 sight and no known ocular issues. Still, I was determined I could ease the strain on my eyes.

Alternatives to solarized

There are too many to list, but one which caught my eye was gruvbox. Specifically, gruvbox has a pretty warm palette and minimal emphasis on blue, which is quite a change from my old setup. Here are some comparison images.

Adopting gruvbox everywhere

gruvbox itself was originally made as just a Vim colorscheme. In its official git repository, you won’t find support for any other programs. However, there is a gruvbox-contrib repository which has support for all sorts of themes for editors, shells, and GUI programs. I’m using the following:

Further blue light reduction

To take things a step further, I also installed Redshift, which globally affects the warmth of the screen’s colors based on the time of day. That is, as the sun goes down, Redshift will gradually make the screen have warmer colors, with the least amount of blue at night. Redshift is conveniently in the official Arch repositories and it works out of the box. I just added the following to my ~/.xinitrc:

# Remove blue light at night
redshift-gtk &

Verdict

Has gruvbox with Redshift actually helped? When I’m using the computer, especially at night, my eyes do still feel strained. However, if I disable Redshift and switch solarized dark back on, it’s very tough to look at the screen again! There’s such a stark difference between looking at gruvbox + Redshift and looking at solarized dark. With that said, I consider this decision a definite win. I’d also note that gruvbox hasn’t seemed any more difficult to view than solarized during the day as well.

Surprisingly, Redshift works well even while playing games on Steam. While playing Rocket League at night, for example, Redshift warms up so much of the blue that it’s shocking when I temporarily disable it to compare.

As for the remaining eye strain, perhaps it’s best to just finish computing earlier at night and schedule more time for reading.

Don’t abandon Mozilla Firefox just yet

2017-12-16T00:00:00-08:00

Just today, a diatribe against Mozilla made it to the front page of Hacker News, gathering a great deal of discussion. Unfortunately, a significant portion of that discussion was around choosing alternatives to Firefox, forking it, or otherwise abandoning ship. Please, if you do care about your privacy, security, and voice in the matter, refrain from the brash decisions and read on.

Some of the key issues

Briefly, some of the issues being discussed are the uninvited introduction of Firefox Pocket, as well as the side-loading of plugins by Mozilla, used in aid of promotion of a TV show. The effect of this promotion is mostly benign, but the problem is that this was done entirely unbeknownst to the user and was enabled by default.

So, time for a new browser

This is the next logical step for a number of participants thus far, but I think they’re not quite considering the implications of using those browsers, as well as the possibility that Mozilla is not a completely lost cause. First, a bit on forks and soft forks, as well as Chrome and Chromium.

Chrome and Chromium

Perhaps the worst result of this assault is that Firefox users will jump over to Chrome. Put simply, since the issue here is about privacy and control over one’s browser, that makes absolutely no sense. This is a serious problem with diatribes against Mozilla for something so relatively small; compared to Google, and the lack of privacy you have within its ecosystem, these Mozilla nitpicks are bantam.

Even with Chromium, there are still in-built Google services which impose upon your privacy. This has lead to the soft fork ungoogled-chromium, but forks have their own baggage to carry as well.

Forks of Firefox or Chromium

To start with, note that browsers are one of the most complex programs running on your OS. Interfacing with the internet, various sites, technologies, etc. means that browsers need to be constantly updating to remain secure. Running any version of Firefox, for example, which is not either the latest or the latest ESR, is putting yourself at risk of all of the known exploits which have been found since.

Hard forks are at serious risk of growing stale behind the upstream parent. Furthermore, any hard fork with a focus on privacy and security will need its own team of developers specifically looking for and quickly patching vulnerabilities in both the new code, as well as the originally forked code. Mozilla pays its security engineers over $150K per year to focus on this sort of work full-time. Replicating that level of commitment with a hard fork is incredibly difficult.

Soft forks

A generally safer bet is a soft fork, which follows the upstream parent closely, but typically just removes some features, or makes some relatively small changes which allow it to remain quite compatible with its parent. Examples of these would be Waterfox and the previously linked ungoogled-chromium. While it’s easier for these projects to follow the progress of the parent, it’s still not guaranteed. Furthermore, the removal of some feature in the soft fork, or the preservation of an old feature which was removed in the parent, may introduce new vulnerabilities which are unique to the soft fork.

Catching these new issues would require security-minded individuals working on the project regularly. Ensuring that the soft fork is following the parent closely remains up to the whim of those running the project. In short, it’s not nearly as reliable as sticking with a company such as Mozilla, who has been invested in developing Firefox for over 15 years.

What to do with Mozilla

So, don’t bite the hand that feeds?

No, that’s not my point. Mozilla shouldn’t have side-loaded plugins in my browser to promote a show of which I’d never even heard. I also still don’t like seeing Pocket there, an uninvited guest in my otherwise tidy browsing environment. Still, that doesn’t mean that Mozilla is evil and I should try my luck with Google or some fork on Github which managed to get a few hundred stars. Instead, it’s important to voice the opinion to Mozilla that this wasn’t cool and it should be removed. Jumping to a one-off fork is not a sustainable option; working with Mozilla, as a community, to show how pissed or pleased we are, at any given moment, is a much more sustainable option.

The Mozilla manifesto contains a few great points relevant to this topic. One of which is the involvement of commercial products in the internet.

Commercial involvement in the development of the Internet brings many benefits; a balance between commercial profit and public benefit is critical.

This balance is hard to strike. It’s the sort of struggle which requires the community to be as active with Mozilla as the companies for which Mozilla is advertising. That’s not to say it’s the community’s fault for this happening; it’s to say that Mozilla isn’t perfect and we occasionally need to slap it back into shape. Right now, it’d be great to see more people herding Mozilla and fewer assuming all is lost.

A guide to using NoScript 10.x

2017-11-30T00:00:00-08:00

When Firefox 57 arrived, NoScript 5 users were left with the unsettling reality of not being able to selectively filter JavaScript until the next version was released, a week or so later. Worse, its release was poorly received, due to bugs and a brand new UI. Now that the dust has settled, and the bugs have been vanquished, NoScript users are still presented with the foreign UI and little to no official documentation. This post will serve as a guide for both old and new NoScript 10 users to get up to speed.

The trust levels

Each domain in NoScript 10 has a trust level. By default, each domain is under the Default trust level, but you may find your old NoScript domains already under the Trusted level. These are familiar to previous NoScript users: the default is not to allow JavaScript, but domains can be explicitly Trusted or Temporarily Trusted. NoScript 10 also adds two more levels: Untrusted and Custom.

Though the UI makes this a bit unclear, these trust levels are columns. For each domain, one button is selected, meaning one trust level.

Editing trust levels

Each trust level can be edited to describe what happens when you set a domain to that level. For Default, Temporarily Trusted, Trusted, and Untrusted, that configuration is global. That is, you you say that the Default trust level should trust fonts (by checking the fonts checkbox), but nothing else, then every domain in that trust level will have that setting. When you modify the Custom trust level, it’s per-domain.

To edit a global trust level, enter the NoScript settings page by clicking the settings button in the NoScript menu.

Once you’re there, you can specify the global properties for each trust level. Note that the properties for Temporarily Trusted reflect whatever you have set for Trusted. The only difference is that Temporarily Trusted domains are only trusted for the current session.

Safe defaults

NoScript 10 allows for more control over what’s filtered. Scripts, objects, media, frames, fonts, WebGL, fetch, and other things can all be filtered. Since you’re using NoScript for privacy, I recommend changing your Default trust level to uncheck all items. If you trust a domain completely, add it to the Trusted level. If you only want some items from it, like fonts and WebGL, then use a Custom trust level for that domain.

Top-level domain configuration

When browsing sites and using NoScript 10, you’ll likely notice that a domain will show up multiple times within the NoScript menu. This is because NoScript 10 allows you to modify the trust level of the specific domain used (such as blog.jeaye.com), as well as the entire top-level domain (such as …jeaye.com). If you set a top-level domain to a specific trust level, that trust level will apply to all sub-domains as well. This is very handy for marking entire ad/tracking domains as Untrusted.

Permanent trust levels

When getting a site to work, I recommend only marking the domains as Temporarily Trusted. Once the site is in working condition, you can upgrade them to permanently Trusted. If you don’t plan on returning to the site often, consider leaving the trust level temporary.

Trusting HTTPS only

NoScript 10 provides the ability to trust a domain only if it’s through a secure connection. You can tell if this is enabled by whether or not the lock, which appears on that domain’s row, is green or red. If it’s red, then the domain will be trusted even through unencrypted connections. This is a concern, since unencrypted connections are vulnerable to MITM attacks and you may be trusting arbitrary and nefarious JavaScript. I recommend making sure the lock is always green (you can click on the lock to toggle it).

Note: If the site only works through HTTP, then trusting HTTPS only will effectively change the site’s trust level back to Default. You may need to keep the lock red for trusted sites which only use HTTP, but you should think twice about trusting anything over HTTP.

For previous NoScript users, I also recommend going back through your whole trusted list and setting all domains to have a green lock. You can get to that list by clicking the options button within the NoScript menu. There isn’t an automatic process for marking them all green yet, but it only took me 10 minutes or so to both ensure all locks are green and do some cleaning of old domains I no longer need to trust.

The remaining UI

With all of that covered, the only two remaining buttons are for revoking all temporary permissions and temporarily trusting the whole page. You can mouse over each of them to see a tool tip for what they do; it’s analogous to the previous NoScript’s behavior. Finally, at this point, you know all you need to effectively use NoScript 10. Don’t leave home without it!

The five common forms of Clojure keywords

2017-10-31T00:00:00-07:00

Depending on which libraries are being used, Clojure has a handful of various idiomatic forms keywords can take. When approaching some forms, like those in Datomic, the overall intention may not be immediately clear. For a new Clojure developer, it may also be unclear which form should be the default, and why. This post aims to add some clarity to the subject and it applies to both Clojure and ClojureScript. Along with explanations of each keyword form is a recommendation for when to use it and when to opt for something else.

Brief: the five common forms

:foo, which is just your plain old keyword
::foo, which is a namespaced keyword for the current namespace
:my.ns/name, which is a namespaced keyword for a valid namespace
::my/name, which uses the :as alias to achieve the same as form #3
:something/foo, which is commonly used with Datomic and doesn’t actually map to a valid namespace

Plain old keywords

These will show up most often in the Clojure and ClojureScript available on the web. They’re easy to type, they don’t require any dependencies, and they make it easy for anyone to consume. That convenience, however, comes at a cost; they can easily cause name collisions, they don’t convey ownership, and, for those reasons, they can’t be used to name specs with clojure.spec.

Example

(def my-data {:x 1.5 :y 0.0})

Recommendation

Avoid plain old keywords by default. If you have a map of data being passed around, for example, namespace the keywords (and consider adding specs for the data). If you have an “enum,” meaning one in a discrete set of possible keywords, also namespace them. The only time when a plain old keyword’s convenience overcomes its cost is within a simple API or DSL with no middleware, so no possibility for collisions. Examples of this would be:

; Simple keyword arguments.
(json/read-str "{}" :key-fn keyword)

(s/keys :req [])

Namespaced keywords

This applies to forms #2, #3, and #4 specifically. These are necessary for specs. They convey ownership, since they’re tied to a valid namespace, they completely avoid the issue of name collision, and they can help explicitly spell out dependencies. Though they may feel like extra work, since you will need to treat them as dependencies, willy-nilly access to data is not a good thing and being explicit about ownership is.

Example

; Form 2.
(ns my.vector)
(s/def ::x number?)
(s/def ::y number?)
(s/def ::2d (s/keys :req [::x ::y]))

; Form 4.
(ns my.game-object
  (:require [my.vector :as v]))
(s/def ::position ::v/2d)
(def my-data {::v/x 1.5 ::v/y 0.0})

; Form 3. Some declaractive config or scene.
{:my.game/objects [{:my.game-object/id :coltrane
                    :my.game-object/position {:my.vector/x 5.0
                                              :my.vector/y 1.0}}]}

Recommendation

Forms #2 and #4 should be your default in Clojure and ClojureScript. Within the same namespace, ::foo is only one more character than :foo, but it carries significantly more data. When you want to access some other system’s data from your app state, for example, you have a dependency on that data. Tying that dependency on a namespace level, through a (:require [my.ns :as my-ns]) allows you to then use the shorthand #4 form ::my-ns/foo. If you detect cyclical dependencies and can’t reorganize, or you need to avoid the require for another reason, then the #3 form can be used. Similarly, within your config.edn, or similar, you’ll use form #3, since you likely have no requires.

Grouped keywords

Grouped keywords (my own terminology) match form #5. Syntactically, grouped keywords are namespaced keywords, but they’re not tied to a valid namespace. Instead, the namespace segment is used for some logical grouping. Datomic uses this for grouping attributes, like :db/id and :user/name. Note that grouped keywords may also use nested group names, like :db.type/long. These don’t map to valid namespaces.

Example

(def order-schema
  [{:db/ident :order/items
    :db/valueType :db.type/ref
    :db/cardinality :db.cardinality/many
    :db/isComponent true}
   {:db/ident :item/id
    :db/valueType :db.type/ref
    :db/cardinality :db.cardinality/one}
   {:db/ident :item/count
    :db/valueType :db.type/long
    :db/cardinality :db.cardinality/one}])
(d/transact conn {:tx-data order-schema})

Recommendation

Avoid these in most situations, but use them where idiomatic. Given that you may want specs for these keywords anyway, I would recommend replacing :db/id with ::db/id and building a my-app.db namespace with the correct specs; use your own good judgement.

Dotted keywords

Finally, some Clojure libraries allow, or encourage, the use of dotted keywords (my own terminology). Dotted keywords are plain old keywords, but they’re specifically used for string building. They’re somewhat more convenient than using strings, since keywords just have a prefix and needn’t be enclosed in quotes. They are less common and aren’t necessarily recommended, but, within a DSL, they can feel quite natural. Due to their uncommonness, dotted keywords aren’t included in the five common forms; they’re included here as an honorable mention.

Example: HoneySQL

HoneySQL is an excellent query builder for numerous SQL databases.

; Keywords like :f.a are used for string building.
(-> {:select [:a :b :c]
     :from [:foo]
     :where [:= :f.a "baz"]}
    sql/format)
=> ["SELECT a, b, c FROM foo WHERE f.a = ?" "baz"]

; Keywords are also used with symbols to convey operations.
(-> (select :*)
    (from :foo)
    (where [:= :a 1] [:< :b 100])
    sql/format)
=> ["SELECT * FROM foo WHERE (a = ? AND b < ?)" 1 100]

Example: cljs-oops

cljs-oops is an essential library for any ClojureScript being compiled with :advanced optimizations.

; Keywords can be used for accessing members of JS objects.
(oget my-js-obj :my-member)

; Nested members can be accessed using dotted keywords.
(oget my-js-obj :transform.position.x)

; Calling functions is much the same.
(ocall js/Math :abs my-debt)

Summary

Keywords in Clojure are certainly versatile. This post has covered plain old keywords, three forms of namespaced keywords, grouped keywords, and dotted keywords. You should now also know their common uses and pitfalls. When in doubt, namespace your keywords to convey explicit ownership of data and prevent name collisions. When building a DSL without middleware or when using keyword arguments, plain old keywords will probably do. Where possible, use clojure.spec!

Async/await in ClojureScript with Promesa

2017-09-30T00:00:00-07:00

JavaScript (ES7/ES2016) introduced async and await as a clean way of working with promises. When porting this sort of asynchronous code to ClojureScript, it may be disappointing that there’s no equivalent language feature. Some may say “use core.async,” which is a fine suggestion, but it may not work with the existing JS promises/thenables. This is where promesa comes in. promesa is a Clojure/Script library for working with native promises; it also provides macro support for async/await and uses core.async machinery behind the scenes.

The JavaScript

This example references Appium and webdriver.io since they’re what I was using when initially researching promesa. This isn’t the best example usage of await, but it allows the code to read as though it works synchronously, even though each line is awaiting a promise.

async function sign_in() {
  await driver.init().timeoutsImplicitWait(120 * 1000);

  await driver.waitForVisible("~email");
  await driver.element("~email").setValue("test@example.com");
  await driver.click("~sign-in");

  await driver.pause(1000).end();
}

Awaiting a JS promise

The first attempt won’t be the cleanest, but it’ll mimic the functionality. In order to turn a JS promise, or, more correctly, any JS thenable, into a bluebird promise (which is what promesa uses behind the scenes), one needs to call js/Promise.resolve. After resolving, the thenable will be a bluebird promise and all of the niceties of promesa will be available.

(ns my-app.test.sign-in
  (:require [[oops.core :refer [ocall]]
             [promesa.core :as p]
             [promesa.async-cljs :refer-macros [async]]]))

(defn sign-in []
  ; async is a promesa macro to allow for awaiting promises.
  (async
    (-> driver
        (ocall :init) (ocall :timeoutsImplicitWait (* 120 1000))
        ; We resolve the thenable into a bluebird promise...
        js/Promise.resolve
        ; Then we await the promise, just like in the JS version.
        p/await)

    ; The repeated resolve and await are an eye sore, but we can't just put them
    ; into a fn, since await can only be used within the async form. We need a
    ; macro.
    (-> driver
        (ocall :waitForVisible "~email")
        js/Promise.resolve
        p/await)

    (-> driver
        (ocall :element "~email") (ocall :setValue "test@example.com")
        js/Promise.resolve
        p/await)

    (-> driver
        (ocall :click "~sign-in")
        js/Promise.resolve
        p/await)

    (-> driver
        (ocall :pause 1000)
        (ocall :end)
        js/Promise.resolve
        p/await)))

Removing some redundancy

Cleaning up the duplication can be done with a simple await-> macro.

; This must be a cljc file.
(ns my-app.test.util.macro
  (:require [promesa.core :as p]))

#?(:clj
   (defmacro await-> [thenable & thens]
     `(-> ~thenable
          ~@thens
          ~'js/Promise.resolve
          p/await)))

(ns my-app.test.sign-in
  (:require [oops.core :refer [ocall]]
            [promesa.core :as p]
            [promesa.async-cljs :refer-macros [async]]
            [my-app.test.util.macro :refer-macros [await->]]))

(defn sign-in []
  (async
    (await-> driver
             (ocall :init) (ocall :timeoutsImplicitWait (* 120 1000)))

    (await-> driver
             (ocall :waitForVisible "~email"))

    (await-> driver
             (ocall :element "~email") (ocall :setValue "test@example.com"))

    (await-> driver
             (ocall :click "~sign-in"))

    (await-> driver
             (ocall :pause 1000)
             (ocall :end))))

Sticking with core.async

For those interested in sticking with core.async, perhaps due to an existing dependency, there is cljs-promises, which allows the conversion of JS promises into async channels. Unlike bluebird’s resolve, cljs-promises doesn’t work with all thenables though, so it’s not as flexible as promesa in that regard.

The state of code quality tools in Clojure

2017-08-31T00:00:00-07:00

There are several projects for verifying quality and correctness in Clojure code bases. When surveying how they’ll work on a distributed Clojure server built for Heroku & Postgres, I took some notes on the issues that came up. These notes represent the current state of things, as well as how we, the Clojure community, can help move them forward.

Out of the nine projects evaluated, only three of them worked, and only two were immediately useful for continuous integration (CI). It’s worth noting that the code base being tested is nothing special, in terms of features used, but it does contain cljc files shared between Clojure and ClojureScript and it makes heavy use of spec and namespace-qualified keywords. Apparently this causes a lot of issues with tooling!

Project: eastwood

[jonase/eastwood "0.2.4"]

Eastwood is a great looking linter for Clojure. It’s designed to find all sorts of issues and code smells, but it’s unfortunately unable to run on the server code, since it doesn’t support namespaced map literals. In the namespaces where it does run, it raises the same false-positive over and over.

Relevant tickets

Project: yagni

[venantius/yagni "0.1.4"]

yagni is an acronym for You Aren’t Gonna Need It and the project is a dead code finder (the only one of its kind, in the Clojure world). Unfortunately, it doesn’t support reader conditionals and it also chokes on the usage of spec.

Relevant tickets

Project: kibit

[lein-kibit "0.1.5"]

kibit is an analyzer which uses core.logic to search for patterns in code which can be written to be more idiomatic. Like many other tools on this list, kibit struggles with parsing some Clojure, like nested requires and reader conditionals.

Relevant tickets

Project: lein-bikeshed

[lein-bikeshed "0.4.1"]

lein-bikeshed is one of the few tools which worked out of the box. Unfortunately, the output wasn’t terribly useful. The two issues reported were that some lines are longer than 80 characters and some functions missing doc strings. I think, given its name, that lein-bikeshed doesn’t take itself too seriously in providing the most practical tooling for CI integration, but it certainly gets points for not choking on the code.

Project: slamhound

[slamhound "1.5.5"]

By only comparing the descriptions, slamhound seemed like one of the most useful ones of the bunch. It has the ability to clean up namespace aliases, add requires, and even remove unused requires.

Unfortunately, it hasn’t been maintained for a year and has started sprouting various issues and PRs which go unloved. More importantly, it fails to parse aliased keywords, as well as the parsing of dotted namespace aliases.

Relevant tickets

Project: spectrum

Spectrum is the one on this list which is most exciting for use with CI. It performs static analysis and, using both specs and type inference, determines if code is incorrect without running it. Given how robust specs can be, reaching into the land of dependent types, this is a very promising avenue. Unfortunately, after speaking with the developer for a bit, it’s clear that spectrum isn’t yet ready to use.

In preparation for it, spec your functions!

Project: lein-nvd

[lein-nvd "0.3.0"]

This tool is a must. Furthermore, it worked without issue. It crawls through a project’s dependencies and checks for vulnerable versions of libraries. If there’s a known vulnerability in one of your dependencies, it informs you and also conveys the severity of the issue. I don’t see why a Clojure back-end would run CI without this.

Project: orchestra

[orchestra "2017.08.13"]

Orchestra is a Clojure library made as a drop-in replacement for clojure.spec.test.alpha, which provides custom instrumentation that validates all aspects of function specs. Best of all, it works out of the box on this source code and can be used during testing, development, and even production.

Project: cloverage

[lein-cloverage "1.0.9"]

I’ve written about cloverage before and I’ll recommend it again. cloverage will report how much coverage your tests have by instrumenting the code to see which functions are called, which branches are taken, etc. It works out of the box with this code base and integrates very nicely into a CI setup.

Striking similarities

There are some promising tools out there, just in the Clojure JVM world, for verifying code quality and correctness, automatically making improvements, or even just suggesting cleaner idioms. It seems like most of them, in fact, are struggling with parsing Clojure code, be it reader conditionals, namespace aliases, nested requires, or aliased keywords. Given that this is a consistent problem across several projects and several authors, there is an indication of insufficient or unreliable tooling when it comes to statically reading Clojure code. This is likely due to Clojure’s dynamic nature, but, nevertheless, there’s an untapped market here.

Given all of the tickets linked above, I encourage readers to try out these projects on their code bases, issue more tickets, ping maintainers, and contribute some PRs. Lastly, if there’s a project that should be on this list, email me!

NixOS: A lasting impression

2017-07-30T00:00:00-07:00

Two years ago, I wrote about my first impression of NixOS, as I was using it in my workstation. While I adored the concept of declarative OS configuration, it didn’t quite fit the workflow I had in mind for my laptop. The post was concluded with me considering NixOS for my VPS, but I didn’t quite want to move away from DigitalOcean, which has support for only a few distros. What follows details how I’ve been running NixOS since, what it took, and what I’ve learned.

Dealing with DigitalOcean

My VPS is hosting this blog, jeaye.com, and many other sites and services. For this, I use DigitalOcean. Ever since I started with DigitalOcean a few years ago, they’ve been joy to work with, so I really didn’t want to leave. Alas, they don’t support many distros, and certainly not custom ISOs. The first droplet I was administrating was already going against the grain, running Arch using a script which converts Debian to Arch in place.

“What a neat idea,” I thought, “to trick DigitalOcean into thinking it’s running a supported distro.” A couple weeks later, nixos-in-place was born; it remains the most stable solution for converting any running GNU/Linux setup to NixOS in place.

Setting up complex services

Once NixOS was running on my DigitalOcean droplet, I had the work of porting all of the services I was running on my Arch droplet to a declarative setup which I could easily version with git. Here are some of the services I’m running, as well as the related configs for each.

For the most part, setting up a service on NixOS is similar to setting it up on any normal distro. The difference is that one generally is limited to the API provided by that NixOS service. Unlike packages, in Nix and NixOS, services can’t be overridden. This has only been an issue once, in the past couple years, but it’s currently limiting my ability to configure spamassassin (master has a much better interface than 17.03).

I tend to forget things, like what I’ve set up on a machine, or everything that was required to get a service running, so having it all in plain text, and version control, in a reproducible fashion, is ideal.

Managing user homes

NixOS doesn’t provide a way to declaratively manage user homes. In fact, the only direct control it provides, declaratively, is over what’s in /etc and its subdirectories. There have been some approaches and discussions (like here and here), but I opted for a much simpler solution, which was already supported at the time. NixOS forfeits the conventional ideas of where programs live and how they’re installed and upgraded, so why not take that liberty with user homes?

So, since NixOS provides declarative management of /etc and its subdirectories, I just use /etc/user as my analogous /home. For example, the user jeaye, defined here, is described as:

users.users.jeaye =
{
  isNormalUser = true;
  home = "/etc/user/jeaye";
  createHome = true;
  extraGroups = [ "wheel" ];
};

If I want to put anything in the home directory for jeaye, I can do so declaratively, like so:

environment.etc =
{
  "user/jeaye/.procmailrc" =
  {
    text =
    ''
      INCLUDERC=/etc/user/jeaye/.procmail/list.rc
      INCLUDERC=/etc/user/jeaye/.procmail/work.rc
      INCLUDERC=/etc/user/jeaye/.procmail/admin.rc
    '';
  };
};

I also use a trick to ensure directories exist, which just involves declaratively putting a hidden file in there. NixOS will create any parent directories needed.

# Ensure that the /etc/user/safepaste/paste directory exists
environment.etc."user/safepaste/paste/.manage-directory".text = "";

Sticking with mainstream configuration to avoid compilations

Initially, I was amazed that my VPS was regularly spending an hour compiling OpenJDK every time I updated. After further investigation, in the helpful #nixos IRC channel on Freenode, it seems this is because I had disabled X everywhere I could. On a headless server, this seemed intuitive. Unfortunately, NixOS Hydra, which builds all of NixOS’ deterministic binaries, only builds with so many configurations. As one can imagine, each new configuration added for a build, with the various platforms, architectures, and other configurations, expands the build time exponentially. As such, the VPS now runs with environment.noXlibs set to false.

Hitting the network in an activation script

I had an upgrade script, which I was running in system.activationScripts, that hit the network to check for upgrades. While the machine was already running, this posed no problem at all and worked quite nicely. Whenever I would nixos-rebuild switch, the activation script would run and upgrade the package if needed. Alas, during a routine reboot, I was no longer able to boot even into a shell; the kernel would panic. After several hours of painstaking debugging with a custom initrd, it turns out the problem was that the activation script hitting the network was failing, since there was no network, and the rest of the boot would then fail.

In short, leave network IO out of activation scripts; I’m using a cron job, for this task, instead. Simple enough.

Building Clojure packages with Leiningen

When bringing in some of my Clojure services, there were issues with compiling Leiningen projects, due to the dependency downloading. By default, the home of the Nix builder isn’t writable, so some workarounds are needed. This setup has been working for me (as part of your typical Nix package):

buildInputs = [ pkgs.leiningen ];
buildPhase =
''
  # For leiningen
  export HOME=$PWD
  export LEIN_HOME=$HOME/.lein
  mkdir -p $LEIN_HOME
  echo "{:user {:local-repo \"$LEIN_HOME\"}}" > $LEIN_HOME/profiles.clj

  ${pkgs.leiningen}/bin/lein uberjar
'';

Pulling from unstable when packages are too old

Occasionally, a package in the Nix repos for a given release, like 17.03, will be too old, have a bug, etc. If NixOS master has a fix for this, it might be worthwhile to bring in the master version of just that package, not your whole OS. Due to the elegance of Nix’s dependency management, this isn’t a problem at all; the unstable channel follows each successful build of the master branch, so we can just pull that in. Say we wanted to do this for weechat.

environment.systemPackages = let pkgsUnstable = import
(
  fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/nixos-unstable.tar.gz
)
{ };
in
[
  pkgsUnstable.weechat
];

Once the next version has been released, or the fix has been added to your default channel, then this can go back to normal.

environment.systemPackages =
[
  pkgs.weechat
];

Minimizing used disk space

If left on its own, NixOS can be quite greedy with disk space. This is primarily a trade-off for the convenience of a purely functional package infrastructure, but it’s still worth noting how it can be managed. The following bits have helped keep my system pretty lean (/nix is 5.4G).

# Auto GC every morning
nix.gc.automatic = false;
services.cron.systemCronJobs = [ "0 3 * * * root /etc/admin/optimize-nix" ];

environment.etc =
{
  "admin/optimize-nix" =
  {
    text =
    ''
      #!/run/current-system/sw/bin/bash
      set -eu

      # Delete everything from this profile that isn't currently needed
      nix-env --delete-generations old

      # Delete generations older than a week
      nix-collect-garbage
      nix-collect-garbage --delete-older-than 7d

      # Optimize
      nix-store --gc --print-dead
      nix-store --optimise
    '';
    mode = "0774";
  };
};

Various nitpicks which could improve the NixOS user’s experience

Calculating SHA-256 of a package isn’t easy

It seems like the best way to do this is to put in a bad SHA-256, try to build, have it fail and tell you the correct one, and then put it in.
Nix command-line UI needs a re-design

This is a known issue but, after nearly two years, has not yet been merged. In short, commands like nix-env -qa would become nix search and nix-env -qc would become nix status. There’s no reason users should have to remember, or learn, the former.
Editor support for Nix and Nixpkgs

Yes, there are Nix plugins for every good editor. What I haven’t seen any formal discussion about, but could help Nix along, is a more functional integration into those editors. I think Emacs is currently best posed, in this regard, but providing semantic completion of all the items in nixpkgs, completion for dependency injection and Nix’s standard library, semantic highlighting, linting, SHA-256 calculation (when writing packages), etc., might really help users jump into the Nix world.
Declaring private data

This is something I’ve yet to tackle. Instead, there are various places within my NixOS files which are marked as XXX, with a comment saying what I need to do. These comments represent imperative steps I need to take when deploying this configuration to a new machine. Currently, this just entails setting up private keys, htpassword files, and some private git repos which I host. A possible solution for passwords, and the like, may be to commit them to git after GPG-encrypting them. For everything else, perhaps a GPG-encrypted bash script in the repo, which does the remaining setup interactively, would suffice.

Considering what’s left and how things are

It’s been two years with NixOS on my VPS and they’ve been great. My biggest complaints are in the form of the Nix expression language itself not being very easy to use, having a good standard library, and having much documentation on doing generic tasks. In this regard, I think that GuixSD is much more appealing: it uses Guile Scheme, which has clear practical applications and is a much more general-purpose language that system administrators might even already know.

Guix’s stance on free software, due to it being a GNU project, is also more appealing; my VPS has absolutely no need for proprietary software (it’s only somewhat harder to argue that for my workstation).

I very much plan to keep NixOS running on my VPS and switching as much as I can to the declarative style. After having such great success in the server world, I’ve been thinking more about trying it again for my workstation. Alas, I think my issues would the Nix language would bug me enough to where that wouldn’t be enjoyable. If I can work out getting Skype (and maybe nVidia) on GuixSD, or maybe if GNU Ring stabilizes enough, then I’d really enjoy having a declarative workspace in very much the same fashion. Maybe in two years I’ll be following up with my thoughts on that.

Porting Orchestra to ClojureScript

2017-06-30T00:00:00-07:00

Orchestra is a Clojure library made as a drop-in replacement for clojure.spec.test.alpha, which provides custom instrumentation that validates all aspects of function specs. Now, it also works with ClojureScript. This post covers some pitfalls of porting Clojure projects to ClojureScript, as well as some instruction for those looking to do so while maintaining a respectable amount of sanity.

Consideration: Directory structure

For sake of illustration, imagine you’re porting a hypothetical kitty-ninja Clojure project to ClojureScript. Your directory structure almost certainly looks like this:

.
├── project.clj
├── src
│   └── kitty_ninja
│       ├── core.clj
│       └── meow.clj
└── test
    └── kitty_ninja
        ├── core_test.clj
        └── meow_test.clj

There are three types of Clojure files to deal with, when supporting both Clojure and ClojureScript. There’s the obvious clj and cljs, but also cljc, which represents a file which may be compiled as either Clojure or ClojureScript. When refactoring your shared code from Clojure land to be exposed to ClojureScript, cljc is what will be used. For the most part, code in cljc files can either be Clojure or ClojureScript, but platform-specific bits can also be included using reader conditionals.

The directory structure is thus changed to something like this:

.
├── project.clj
├── src
│   ├── clj
│   │   └── kitty_ninja
│   │       └── core.clj
│   ├── cljc
│   │   └── kitty_ninja
│   │       └── meow.cljc
│   └── cljs
│       └── kitty_ninja_cljs
│           └── core.cljs
└── test
    ├── clj
    │   └── kitty_ninja
    │       └── core_test.clj
    ├── cljc
    │   └── kitty_ninja
    │       └── meow_test.cljc
    └── cljs
        └── kitty_ninja_cljs
            └── core_test.cljs

As is shown, the meow functionality is shared between both Clojure and ClojureScript. Furthermore, its tests are also shared. Aside from that, both Clojure and ClojureScript have their own platform-specific code and tests/entry points as well. More on this later.

Pitfall: Namespaces

The likely first thought, when looking to port the kitty-ninja project to ClojureScript, would be to also have the ClojureScript counterpart be in the kitty-ninja namespace.

This will cause much, much more harm than good.

For this reason, we have cljs.spec instead of clojure.spec for ClojureScript. Similarly, we now have orchestra-cljs.spec.test instead of orchestra.spec.test. Since ClojureScript can actually require Clojure files, and all of its macros are run in Clojure, as well as the class path issues that arise with having src/clj/ and src/cljs/ in your Leiningen :source-paths, just agree from the start that your ClojureScript port will be in a different namespace.

Pitfall: Macros

In ClojureScript, macro expansion is a very distinct phase of compilation, which involves running Clojure on the ClojureScript code. As such, macros need to be in either a clj or cljc file. Part of refactoring kitty-ninja, or any other project, would involve moving the macros which will be shared with ClojureScript to the src/cljc/ directory. Furthermore, any ClojureScript-specific macros may be kept in the src/cljs/ directory. This may be counter-intuitive, since it means there will be cljc files in src/cljs/, but this makes sense in the case where they’re for ClojureScript-only macros.

There is also a pattern, which has largely gone unstated as far I can tell, where a cljs and cljc file can have the same file name and namespace. If the cljs file requires the cljc version, to bring in its macros, then they’ll automatically be available to anyone to requires the cljs file. Here’s an illustration of how that works:

.
└── src
    └── cljs
        └── kitty_ninja_cljs
            ├── core.cljs
            ├── stealth.cljc
            └── stealth.cljs

stealth.cljc:

(ns kitty-ninja-cljs.stealth)

(defmacro purr []
  `(println "purrrr"))

stealth.cljs:

(ns kitty-ninja-cljs.stealth
  (:require-macros [kitty-ninja-cljs.stealth :as st]))

core.cljs:

(ns kitty-ninja-cljs.core
  (:require [kitty-ninja-cljs.stealth :as st]))

(st/purr) ; Macro invocation without having to do require-macros

Pitfall: Cljsbuild profiles

Unlike Leiningen profiles, which are quite flexible, cljsbuild profiles tend toward redundancy. Furthermore, cljsbuild requires a top-level key in the project.clj. As a result, the best setup I’ve found is to provide the base information at the top-level, using a fixed build name, rather than in a shared Leiningen profile. From there, use Leiningen profiles to customize the values you need. Here’s an annotated example from Orchestra:

(defproject ; ... elided a great deal ...

  ; Keep cljs in here so `lein jar` packages the sources
  :source-paths ["src/clj/" "src/cljs/"]

  ; Base cljs setup
  :cljsbuild {:test-commands {"test" ["lein" "doo" "node" "app" "once"]}
              :builds {:app
                       {:source-paths ["src/cljs/"]
                        :compiler
                        {:optimizations :advanced
                         :pretty-print false
                         :parallel-build true
                         :output-dir "target/test"
                         :output-to "target/test.js"}}}}

  ; Custom dev dependencies, sources, and entry point for test running
  :profiles {:dev {:dependencies [[lein-doo "0.1.7"]]
                   :source-paths ["test/clj/" "test/cljc/"]
                   :cljsbuild {:builds {:app
                                        {:source-paths ["test/cljs/" "test/cljc/"]
                                         :compiler
                                         {:main orchestra-cljs.test
                                          :target :nodejs}}}}}})

Consideration: Tests with doo

A combination of doo and ClojureScript’s cljs.test will allow for mostly painless sharing of tests between Clojure and ClojureScript. A word of caution, however, based on my experience:

Use Node for your tests, not Phantom.

There exists an issue on doo for this, but PhantomJS is not as well supported and, in my experience, not nearly as reliable. Fortunately, switching to Node should be as easy as adding :target :nodejs to the cljsbuild profile and changing the lein doo command to use node instead of phantom. See the Orchestra profile above for reference.

Pitfall: Figwheel’s REPL

If you end up using figwheel, which arguably most non-trivial ClojureScript projects do, absolutely consider using rlwrap (available on most Unix-like platforms). By default, fighwheel’s REPL doesn’t provide history (up and down arrows), word/line based editing (like ^W and ^U), or reverse search of previous inputs (using ^R). rlwrap will provide all of these otherwise ubiquitous features for you, without breaking any figwheel functionality.

Wrapping up

ClojureScript’s tooling doesn’t compare to Clojure’s, but it’s certainly a capable platform. For those already working in ClojureScript, be it with React(Native) or anything else, please do consider using spec and instrumentation to aid in your development. For those coming to ClojureScript, from Clojure, hopefully these points will save you some time.

Dear Clojure devs, use clojure.spec please

2017-05-31T00:00:00-07:00

Clojure 1.9 has introduced a novel library, clojure.spec, which allows developers to parse and validate data using a predicative API which is just composed of Clojure functions. If there’s one thing Clojure’s good at, though there’s not just one, it’s the pure transformation of data. With spec, that grasp is broadened to allow for exceedingly expressive validation of data as well. What’s most important is that Clojure developers understand this new tool and use it to improve the safety and readability of their code.

Learning clojure.spec

It’s been around long enough for there to be some great introductory material. There’s an official Rationale and Overview, thorough Spec Guide, more illustrative Video Introduction, etc. This post assumes a working understanding of spec, since its goal is to talk more about spec’s practical applications.

Why spec?

Knowing how spec works and even how to use it is handy. Still, why bother spec’ing? If you look at the spec docs linked above, most reasons are for the aid in testing, specifically with generative testing. That’s great, if you like writing tests, or if your tests aren’t already written in something else. What if that’s not the case?

Please. Still use clojure.spec.

Here’s why.

Instrumentation

A hugely overlooked aspect of spec is its instrumentation. In short, if you spec your functions and your data, you can ask Clojure to automatically check every single function call to make sure the arguments are correct. Not just during testing, but during development or even production. Furthermore, if you use Orchestra, then Clojure can automatically check every function’s return value against its spec, among other things. Note, this includes ClojureScript!

This is superb.

Coming from C++, or Java, or C#, or so many other languages: have you been able to easily instrument every single function to validate it’s working properly? Not just with static types, which can be quite limiting, but with arbitrary predicates written in the same language you’re using? Unlikely.

Let’s look at some code.

Simple math

(defn my-inc [x]
  (inc x))

This is a simple function, so it’s easy to tell what x needs to be. In more complex functions, that’s not always the case. Now, what if we were to call this with something other than a number?

(my-inc nil) ; => java.lang.NullPointerException
(my-inc "ok") ; => java.lang.ClassCastException

Those are pretty helpful. The exception will include a stack trace, so you can get the line number and find the right function. Let’s spec out that function and see what we’d get with instrumentation enabled.

(require '[clojure.spec.alpha :as s])

(defn my-inc [x]
  (inc x))
(s/fdef my-inc
      :args (s/cat :x number?)
      :ret number?)

Just a single example should be descriptive enough, once we enable Orchestra.

user=> (my-inc "ok")
ExceptionInfo: Call to #'user/my-inc did not conform to spec:
               In: [0]
               val: "ok"
               fails at: [:args :x]
               predicate: number?
               :clojure.spec.alpha/spec #object[...]
               :clojure.spec.alpha/value ("ok")
               :clojure.spec.alpha/args ("ok")
               :clojure.spec.alpha/failure :instrument
               :orchestra.spec.test/caller {:file "form-...",
                                            :line 1,
                                            :var-scope user/eval42203}

That’s gorgeous. Before we get into the function, we’re stopped with some very detailed information that [:args :x] (the argument called x) was supposed to match number?, but it has the value of "ok". We also get to see all the other args to the function, line/file info, etc. Compared to typical statically-typed languages, and the typical Clojure exceptions, which say “expected number, got string,” we’re now dealing with values, not just types. In this way, spec behaves more like a dependent type system.

The most important aspect has yet to be mentioned: if a Clojure library provides these specs for its functions and data, any consumers using Orchestra and spec will immediately be able to benefit from automatic instrumentation. Each spec is also a form of documentation which must be up-to-date with the code (or instrumentation would fail!).

More complex map extraction

; This behaves similarly to clojure.core/select-keys
(defn extract
  "Given a map and some keys, return a map with only those keys"
  [m ks]
  (reduce (fn [acc k]
            (let [v (get m k)]
              (if (some? v)
                (assoc acc k v)
                acc)))
          {}
          ks))
(s/fdef extract
        :args (s/cat :m map?
                     :ks (s/coll-of any?))
        :fn (fn [ctx]
              (= (into #{} (-> ctx :args :ks))
                 (into #{} (-> ctx :ret keys))))
        :ret map?)

In this example, we can use Orchestra to take advantage of spec’s :fn spec. These are executed at the end of a function call, and they’re given both the arguments and the return value. In this case, we can verify that the keys of the output map are exactly the keys meant to be extracted. Note, this expects that all keys were present, but that’s the sort of control which you can embed and automatically run on each function call.

If we intentionally add a bug, where we use if-let instead of if-some or some?, this function won’t work well with false.

(defn extract
  "Given a map and some keys, return a map with only those keys"
  [m ks]
  (reduce (fn [acc k]
            (if-let [v (get m k)]
              (assoc acc k v)
              acc))
          {}
          ks))

With instrumentation enabled, here’s what we might see. Since it’s the :fn spec which fails, the error will give us all of the argument values, as well as the return value. It gives use the predicate which failed and dropping them all in the REPL would allow us to figure out exactly why.

user=> (extract {:foo 0 :bar false :spam "meow"} [:bar])

ExceptionInfo: Call to #'user/extract did not conform to spec:
               val: {:ret {},
                     :args {:m {:foo 0, :bar false, :spam "meow"},
                            :ks [:bar]}}
               fails at: [:fn]
               predicate: (fn [ctx]
                            (= (into #{} (-> ctx :args :ks))
                               (into #{} (-> ctx :ret keys))))
               :clojure.spec.alpha/spec  #object[...]
               :clojure.spec.alpha/value {:ret {},
                                          :args {:m {:foo 0, :bar false, :spam "meow"},
                                                 :ks [:bar]}}
               :clojure.spec.alpha/fn {:ret {},
                                       :args {:m {:foo 0, :bar false, :spam "meow"},
                                              :ks [:bar]}}
               :clojure.spec.alpha/failure :instrument
               :orchestra.spec.test/caller {:file "form-...",
                                            :line 1,
                                            :var-scope user/eval53563}

A real-world example

Just recently, I opened a pull request to Reagent, which is an excellent project, in hopes of improving its input validation and error messages. Rather than using spec, it’s performing manual asserts on input data and then providing adhoc error messages on failed validation. My PR keeps the asserts, but refactors them to common helpers and improves the messages (introducing new deps isn’t typically the right first step in improving a library as an outsider). Still, we can do so much better than that. These can be checked for us and we can describe the shape of the data as it should be when it flows through our Clojure machines.

Performance implications

Automatically instrumenting every single function call, checking all the arguments, return values, and possibly :fn specs sounds pretty slow, right? You may be surprised. For development, I’ve seen absolutely no notable performance issues running Orchestra and instrumenting just about every function in a back-end deployment of Ring + Compojure + PostgreSQL. Your mileage may vary, but this is something you should try first, get as much out of it as you can, and only put down if you absolutely must.

For the safety of your programs and the programs of everyone using your libraries, Clojure devs, please spec out your functions and your data. If you want help writing your specs, heck, email me and let’s get it done. Clojure devs deserve the huge win of automatic instrumentation.