It’s Day 5 of Notarization Week and it’s time to wrap up and write down my experiences.

Notarization itself is not incredibly difficult. You can learn the basics by watching the 40 minutes talk from WWDC 2019. Unlike sandboxing, notarization should not have any detrimental effects for most Mac apps.

As always the real trouble starts when you are trying to inject Notarization into the tangled web of modern Mac software development: entitlements, certificates, automated Xcode build chains, build settings, etc..

First you need to adopt the “Hardened Runtime” for your application. For the two apps that I tested with, this was simply a matter of switching it on in the “Capabilities” tab of your target. By default, all the hardened runtime features are switched on and I was able to leave them all on without any problem.

The first gotcha is that you can’t really test your application’s compatibility with the hardened runtime in Xcode, because it will run in debug mode. Since the hardened runtime would not allow inspection of your code, the default “CODE_SIGN_INJECT_BASE_ENTITLEMENTS=YES” build setting   will inject the “com.apple.security.get-task-allow” entitlement into the debug version of your build product. This is a “normal” entitlement, just like those used for sandboxing.. and no the sandbox does not need to be turned on for notarization to work (sigh of relief).

Another gotcha is that your app will not be notarized as long as this entitlement is switched on, so we need to turn it off for the release build. This should not be a worry, but you will probably spend many frustrating hours chasing down this very problem nonetheless..

The next thing on the compliance list is that secure timestamps for codesign need to be turned on. Many developers have a “–timestamp=none” flag somewhere in their build settings.. because the Apple timestamp servers are slow and often down (at least here in Luxembourg) and you can no longer build a release without an internet connection. So if you have a build server without internet connection.. that is about to change. To make doubly sure, you should probably add “OTHER_CODE_SIGN_FLAGS=’$(inherited)  –timestamp” to your build settings.

In this context, it would have saved me a lot of time if I had known how to find out whether a product has in fact been signed with a secure timestamp. Executing “codesign –verify –deep –strict –verbose=4 –display  -r- /path/to/my/product” will display loads of things. If there is a line with “Signed Time” among it, that means that you did not sign with a secure timestamp. If you have a line with “Timestamp” in it, it means you do have a secure timestamp. It’s another brilliant example of how an Apple engineer’s language choice can cost tens of thousands of lost developer hours. “Signed Time (insecure)” would have been a great help.

In a similar vein, “codesign -d –entitlements :- /path/to/my/product” displays all the entitlements for the product and will reveal the dreaded “com.apple.security.get-task-allow” entitlement if it is still present.

Once you have a build product, you can send it to Apple for notarization with the “xcrun altool –notarize-app -f /my/archive –primary-bundle-id my.primary.bundle.id -u username -p apps-peci-fic-pass-word”.

This is where things get a little weird. You can send either a disk image or a zip archive, but not an application bundle. I distribute my software as disk images and my software updates as zips. If you send a zip file, make sure that you use the “ditto” tool as instructed by Apple, so that you don’t run into problems with extended attributes. You need to supply your username (email address) and a password. You can generate an application specific password and that worked fine for me straight away.

The command line will upload the archive and then return a “request-id” which is a UUID that you can use to look up the state of the notarization. This is not a real-time, synchronous affair. It was fairly quick when I used it, taking usually only a few minutes, but it is obviously a challenging problem for automation. You could write a script that extracts the request-id and then polls the Apple servers for its status before continuing, but realistically you probably want to have a two or three stage build process now.

I subdivided my own build process from one into three phases: build, request notarization and a combined stapling and verification phase.

Which brings us to stapling, which is the fun and easy part. You just type “xcrun stapler staple my.dmg” or “xcrun stapler staple my.app” and that’s that.

One thing to note is that the entire notarization process is completely free of build and version numbers, which is so wonderful. If only app review worked this way! There is no mention on how it works; it could be that Apple uses the entire archive as a hash code or that they create a hash of your upload. In any event, there is zero problem building a thousand different versions of your program and getting them all notarized.

The second thing to notice is that you can either staple app bundles or disk images, but not zip archives. Not sure which is weirder, but it kind of makes sense. In practical terms, this means that you can staple your notarization receipt to a dmg without having to open it, which is super easy. If I have understood this correctly, this means that both the dmg and the app are stapled and will open without any funny user warnings. Not being able to staple zip files, however, complicates things somewhat, because you now have to zip the app bundle to notarize it, staple the original unzipped app bundle and then re-zip it.

So far so good. Now enter the much dreaded Sparkle.framework, the foundation of all automated software updates across the Developer ID world, maintained by a clever, intrepid group of volunteers that deserve our eternal gratitude.. and the bane of my developer life.

For most of my products, Sparkle is the only framework that I bundle, so I blame it for the entire dreaded complexity and wasted time of framework signing.. which is a lot of blame. Signing frameworks is hell.. or used to be hell.. and now is hell again.

I don’t use Carthage or other “download stuff from all over the internet written by who knows who, run buggy build and install scripts and then ship the whole lot with my product” build systems. I actually just place a binary copy of the framework into the /Library/Frameworks/ folder and work with that. If you are using one of those build systems, you probably will have different problems.

The current (as of 26/July/2019) binary distribution of Sparkle is neither signed, nor built with the hardened runtime, so is unusable for notarizated apps. Downloading the source as a zip archive leaves out crucial files. So I did a “git clone –recursive https://github.com/sparkle-project/Sparkle” to get what I assume must be the master branch version (I have some deeply strange git expertise that overlaps with nobody else’s).

Building it with “make release”, despite affirmations to the contrary, did not result in a hardened version. One of worst things (I’m pretty sure it’s unavoidable and I’m not dissing its developers at all, but it is still absolutely dreadful) about Sparkle is that it includes two executables as well as the framework. Autoupdate.app and fileop always cause incredible signing headaches. The default option of just ticking the “Sign upon copy” option in Xcode, won’t sign these properly and you inevitably end up with gatekeeper problems.. even though it had just gone through a phase of actually working.. but no more.

I’m sure that at the heart of all my signing problems is a lack of understanding, aka ignorance. The thing is that I’m a Mac developer, not a cryptography geek. Knowing just enough to get by in the context of cryptography means knowing quite a lot about quite a few things, followed by endless trial and error that eventually ends for unknowable reasons.

After a very long time, I finally got a Sparkle build that I could use by opening the project in Xcode, adding the “OTHER_CODE_SIGN_FLAGS=’$(inherited)  –timestamp”, “CODE_SIGN_INJECT_BASE_ENTITLEMENTS=YES” to every relevant target and manually adding my Developer ID signing identity to all targets. I have no idea why this was necessary; as far as I understand the framework does not need to be signed at all, and will in any event be re-signed when it is copied into my app, but it would not work without this. Perhaps the entitlements only get added during signing?

I then spent most of a day chasing down the origin of the “com.apple.security.get-task-allow” entitlement on the “fileop” executable that steadfastly refused to go away, despite having no debug build and having plastered the “CODE_SIGN_INJECT_BASE_ENTITLEMENTS=YES” build settings everywhere throughout the Sparkle project. Around 11PM, I decided to just delete Xcode’s “Derived” folder (what else was there left?).. and that promptly solved the problem.

With the Sparkle problems solved, the rest was fairly straightforward.

All told I’ve spent an entire week on learning about Notarization and integrating it into my build system.  It’s not badly designed. In fact it works fairly well and I would even go as far as calling some of the design decisions enlightened. It is certainly a lot better thought through than either App Review or the Sandbox.

Unfortunately, it adds yet more complexity to an already incredibly complex development environment. Today’s apps are not much more complex than those from the 1990s. Phone apps are mostly much less complex. It should be easier to develop for the Mac today than it was back in the 1990s. Yet nearly 30 years of development tool, framework and API progress has yielded a development context that is no more productive and far more complex. Notarization adds yet another layer of complexity and yet another time sink for Mac developers.

There are some positives: Apple can now revoke specific builds of an app, rather than just turning off all apps from the same developer id. The hardened runtime gives the developer the possibility of shielding his/her software from malicious modification, but allows him/her to decide which “holes” need to be blasted into runtime for the program to continue working. Actually scanning apps for malware adds peace of mind when you release a program into the world.

In an ideal world, Apple would turn around and ditch its Mac App Store sandbox requirement. It could even offer notarization as a way to side-load software on the iPad. After all, notarization gives it the tools to prevent malware from being published and to switch off on every single Mac in the world should it get through anyway.

As a long time Mac developer (since 1994), however, I can’t help thinking though that the security people at Apple would have done better ironing out the bugs and limitations of the sandbox to get it work properly and be less of a nuisance, rather than adding yet another security approach.

If early reports about Catalina are to be believed, it looks like there are so many people working on Mac security that they have to roll out new security features at each release, whether they are a net benefit to users or not. Perhaps, these people could be tasked with making macOS great again instead?