A series of interiors

I’ve been reading Wanderlust: A History of Walking, and this paragraph made me pause:

Many people nowadays live in a series of interiors — home, car, gym, office, shops — disconnected from each other. On foot everything stays connected, for while walking one occupies the spaces between those interiors in the same way one occupies those interiors. One lives in the whole world rather than in interiors built up against it.

I’ve had a similar reflection before when traveling: that you could leave home and go from taxi to train to airport to the other side of the world and emerge on a subway exit in New York or Tokyo twenty-some hours later, without having set foot on an open space.

And a similar feeling is portrayed early on in Zen and the Art of Motorcycle Maintenance about riding a motorcycle instead of walking:

You see things vacationing on a motorcycle in a way that is completely different from any other. In a car you’re always in a compartment, and because you’re used to it you don’t realize that through that car window everything you see is just more TV. You’re a passive observer and it is all moving by you boringly in a frame.

On a cycle the frame is gone. You’re completely in contact with it all. You’re in the scene, not just watching it anymore, and the sense of presence is overwhelming. That concrete whizzing by five inches below your foot is the real thing, the same stuff you walk on, it’s right there, so blurred you can’t focus on it, yet you can put your foot down and touch it anytime, and the whole thing, the whole experience, is never removed from immediate consciousness.

Diving into Swift compiler performance

It all starts by reading this week in Swift, and the article The best hardware to build with Swift is not what you might think, written by the LinkedIn team about how apparently their Mac Pros are slower at building Swift than any other Mac.

I’ve spent so much time waiting for Xcode to compile over the past years than I’ve often toyed with the idea of getting an iMac or even a Mac Pro for the maximum possible performance, so this caught my attention. I’ve also been wondering if instead of throwing money at the problem, there might be some easy improvements to either reduce build time or to improve Swift performance.

Looking at the reported issue I discovered a couple Swift compiler flags that were new to me: -driver-time-compilation and -Xfrontend -debug-time-compilation, which will show something like this:

===-------------------------------------------------------------------------===
                               Swift compilation
===-------------------------------------------------------------------------===
  Total Execution Time: 10.1296 seconds (10.6736 wall clock)

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
   3.9556 ( 99.9%)   6.1701 (100.0%)  10.1257 (100.0%)  10.6697 (100.0%)  Type checking / Semantic analysis
   0.0013 (  0.0%)   0.0002 (  0.0%)   0.0015 (  0.0%)   0.0015 (  0.0%)  LLVM output
   0.0011 (  0.0%)   0.0001 (  0.0%)   0.0013 (  0.0%)   0.0013 (  0.0%)  SILGen
   0.0005 (  0.0%)   0.0001 (  0.0%)   0.0006 (  0.0%)   0.0006 (  0.0%)  IRGen
   0.0003 (  0.0%)   0.0001 (  0.0%)   0.0003 (  0.0%)   0.0003 (  0.0%)  LLVM optimization
   0.0001 (  0.0%)   0.0001 (  0.0%)   0.0002 (  0.0%)   0.0002 (  0.0%)  Parsing
   0.0000 (  0.0%)   0.0000 (  0.0%)   0.0000 (  0.0%)   0.0000 (  0.0%)  SIL optimization
   0.0000 (  0.0%)   0.0000 (  0.0%)   0.0000 (  0.0%)   0.0000 (  0.0%)  Name binding
   0.0000 (  0.0%)   0.0000 (  0.0%)   0.0000 (  0.0%)   0.0000 (  0.0%)  AST verification
   0.0000 (  0.0%)   0.0000 (  0.0%)   0.0000 (  0.0%)   0.0000 (  0.0%)  SIL verification (pre-optimization)
   0.0000 (  0.0%)   0.0000 (  0.0%)   0.0000 (  0.0%)   0.0000 (  0.0%)  SIL verification (post-optimization)
   3.9589 (100.0%)   6.1707 (100.0%)  10.1296 (100.0%)  10.6736 (100.0%)  Total

I started looking into the results for the WordPress app and it looks like almost every bottleneck is in the Type Checking stage. As I’d find later, mostly in type inference. I’ve only tested this on Debug builds, as that’s where improvements could impact development best. For Release, I’d suspect the optimization stage would take a noticeable amount of time.

Time to look at what’s the slowest thing, and why. Disclaimer: the next shell commands might look extra complicated. I’ve used those tools for over a decade but never managed to fully learn all their power, so I’d jump from grep to awk to sed to cut and back, just because I know how to do it that way. I’m sure there’s a better way, but this got me the results I wanted so ¯\(ツ)/¯.

Before you run these close Xcode, and really everything else if you can so you get more reliable results.

Do a clean build with all the debug flags and save the log. That way you can query it later without having to do another build.

xcodebuild -destination 'platform=iOS Simulator,name=iPhone 7' \
  -sdk iphonesimulator -workspace WordPress.xcworkspace \
  -scheme WordPress -configuration Debug \
  clean build \
  OTHER_SWIFT_FLAGS="-driver-time-compilation \
    -Xfrontend -debug-time-function-bodies \
    -Xfrontend -debug-time-compilation" |
tee profile.log

Print the compiled files sorted by build time:

awk '/Driver Time Compilation/,/Total$/ { print }' profile.log |
  grep compile |
  cut -c 55- |
  sed -e 's/^ *//;s/ (.*%)  compile / /;s/ [^ ]*Bridging-Header.h$//' |
  sed -e "s|$(pwd)/||" |
  sort -rn |
  tee slowest.log

Show the top 10 slowest files:

head -10 slowest.log
2.9555 WordPress/Classes/Extensions/Math.swift
2.8760 WordPress/Classes/Utility/PushAuthenticationManager.swift
2.8751 WordPress/Classes/ViewRelated/Post/AztecPostViewController.swift
2.8748 WordPress/Classes/ViewRelated/People/InvitePersonViewController.swift
2.8741 WordPress/Classes/ViewRelated/System/PagedViewController.swift
2.8699 WordPress/Classes/ViewRelated/Views/WPRichText/WPTextAttachmentManager.swift
2.8680 WordPress/Classes/ViewRelated/Views/PaddedLabel.swift
2.8678 WordPress/Classes/ViewRelated/NUX/WPStyleGuide+NUX.swift
2.8666 WordPress/Classes/Networking/Remote Objects/RemoteSharingButton.swift
2.8162 Pods/Gridicons/Gridicons/Gridicons/GridiconsGenerated.swift

Almost 3 seconds on Math.swift? That doesn’t make any sense. Thanks to the -debug-time-function-bodies flag, I can look into profile.log and see it’s all the round function. To make this easier, and since it doesn’t depend on anything else in the app, I extracted that to a separate file. In this case, the -Xfrontend -debug-time-expression-type-checking flag helped identifying the line where the compiler was spending all the time:

return self + sign * (half - (abs(self) + half) % divisor)

When you look at it, it seems pretty obvious that those are all Ints, right? But what’s obvious to humans, might not be to a compiler. I tried another flag -Xfrontend -debug-constraints which resulted in a 53MB log file 😱. But trying to make sense of it, it became apparent that abs was generic, so the compiler had to guess, and +,-,*, and % had also multiple candidates each, so the type checker seems to go through every combination rating them, before picking a winner. There is some good information on how the type checker works in the Swift repo, but I still have to read that completely.

A simple change (adding as Int) turns the 3 seconds into milliseconds:

return self + sign * (half - (abs(self) as Int + half) % divisor)

I’ve kept going through the list and in many cases I still can’t figure out what is slow, but there were some quick wins there. After 4 simple changes, build time was reduced by 18 seconds, a 12% reduction.

Attaching patches to Pull Requests

This might sound strange, but sometimes I prefer patches to pull requests. The main scenario is when I’m reviewing someone else’s code and I want to propose an alternative implementation.

I could just create a new branch and pull request with my change, but then the conversation is split between two PRs, and there’s a new branch that you have to clean up.

When the change is small enough, or I’m not sure if it will be accepted, I’d rather send a patch. So far I’ve been doing git diff, uploading the result to gist, and posting the link as a comment in the PR. This has a few shortcomings:

  • No binary support.
  • If the original author wants to use it, authorship is usually lost, unless they use the --author option for git commit, and even then there’s room for typos.

I know there’s a better way, as Git was originally designed to share patches, not pull requests. I think I’ve been avoiding it because it’s not as common and the original author might not know what to do with the patch. So I’m writing this as a quick tutorial.

Creating a patch

Before creating a patch, you have to commit your changes. git format-patch will create a patch file for each commit, so your history can be preserved. Once you have a commit, your branch is ahead of origin, so we can use that to tell format-patch which commits to pick

branch=git rev-parse --abbrev-ref HEAD
origin="origin/$branch"
git format-patch $origin

This will leave one or more .patch files in your project directory:

$ ls *.patch
0001-Store-relative-paths-for-reader-topics.patch

Upload those to Gist and leave a comment with the link on the PR:

$ gist -co *.patch

Applying a patch

For a single patch, you can copy the Raw link in the Gist and download it

$ curl -sLO https://gist.github.com/koke/1b30d861e6bb9d366f69bc186d0e9525/raw/8cc27f3e589a7823b2e9f1746aa921b92da14187/0001-Store-relative-paths-for-reader-topics.patch

If there are multiple files, make sure you use the Download Zip link (or download all the files one by one):

$ curl -sLo patches.zip https://gist.github.com/koke/ab100907c17c4ef6a977350494679091/archive/3fb0136a21a6bc499bff2511750c62ae6dc41630.zip
$ unzip -j patches.zip 
Archive:  patches.zip
3fb0136a21a6bc499bff2511750c62ae6dc41630
  inflating: 0001-Store-relative-paths-for-reader-topics.patch  
  inflating: 0002-Whitespace-changes.patch  

Once you have the patch file(s) in your project directory, just run git am -s *.patch:

$ git am -s *.patch
Applying: Store relative paths for reader topics
Applying: Whitespace changes

Review the changes, and if you’re happy with them, git push them. Otherwise, you can reset your branch to point at the pushed changes:

branch=git rev-parse --abbrev-ref HEAD
origin="origin/$branch"
git reset --hard $origin

Finally, run git clean -df, or manually remove the downloaded files.