Mobile Symbolication — Release Gate & Runbook
Owner: Mobile on-call Trigger: mobile-build-native.yml fails on one of the three dSYM/symbolication steps in the build-ios job, a production/TestFlight crash arrives in Sentry with zero stack frames despite this gate, or you are holding a raw .ips crash report and need to symbolicate it by hand. Spec: issue #10831 (the gate) and #11000 (the CI artifact route below), parent epic #10828
In a hurry with a
.ipsin hand and no Sentry access? Skip to Symbolicating a crash report by hand.
Why this exists
The iOS JSI/Hermes launch crash (Sentry issue 122620667) reached 10 re-files and production (#10251, #10418, #10428, #10563, #10649) partly because release builds shipped without their dSYMs ever reaching Sentry. Every occurrence landed with zero stack frames — the team debugged blind for weeks. #10589 added the upload step and #10816 added a local validity check, but both were continue-on-error: a broken upload degraded diagnosability silently and the build still shipped.
#10831 makes this a hard release gate. A build that cannot be symbolicated does not ship, full stop.
An un-symbolicatable crash is a P1 observability incident
If a production or TestFlight native crash arrives in Sentry with no resolved stack trace (raw addresses / <redacted> frames / a single generic frame), treat it the same as any other P1: it means the release pipeline shipped a build this gate should have caught, or Sentry itself is degraded. Do not triage it as "just another unsymbolicated crash" — find out why the gate didn't catch it first; the crash itself is secondary until that's answered.
What the gate actually checks
mobile-build-native.yml's build-ios job runs three steps in sequence, all gated on inputs.submit == true (release/TestFlight-bound builds only — dev/simulator builds never reach this path and legitimately have no dSYMs):
| # | Step | What it proves | Failure mode it closes |
|---|---|---|---|
| 1 | Upload iOS dSYMs to Sentry | sentry-cli debug-files upload exited 0 against the archive's dSYMs/ directory | No dSYMs directory at all (e.g. archive step regressed) |
| 2 | Verify iOS dSYMs are symbolicatable (#10773) | Every local .dSYM bundle is a structurally valid debug file carrying a debug ID (sentry-cli debug-files check) | A stripped/empty dSYM that uploads "successfully" but carries nothing useful (ExportOptions.plist sets stripSwiftSymbols: true on the exported IPA, not the archive dSYMs — but a build misconfiguration could still strip them earlier) |
| 3 | Confirm dSYM symbolication in Sentry (post-upload smoke-check, #10831) | Every dSYM UUID this build just archived (via dwarfdump --uuid) is actually resolvable through the live Sentry API (GET /projects/{org}/{project}/files/dsyms/?query=<uuid>) | Steps 1–2 only prove the upload command succeeded and the local file is valid — neither asks Sentry's server whether the file actually landed and was indexed. This is the gap that let #10649 slip past #10589/#10816. |
All three now run without continue-on-error: any failure fails the build-ios job, which fails the whole mobile-build-native.yml run, which blocks the TestFlight/App Store submission steps later in the same job.
The smoke-check script lives at mobile/ppp/scripts/verify-sentry-symbolication.mjs (unit tests: mobile/ppp/scripts/__tests__/verify-sentry-symbolication.test.mjs, run via pnpm test:scripts). It polls the Sentry API a few times (default 4 attempts, 15s apart) before declaring a UUID missing, to tolerate normal upload-to-index lag rather than false-failing a good upload.
When the gate fails — what to do
- Read the failed step's log and job summary first. Each of the three steps writes a clear
::error::annotation and aGITHUB_STEP_SUMMARYblock naming exactly what's missing (no dSYMs directory / an unvalidatable dSYM / specific UUIDs not resolvable in Sentry). - Check whether this is a Sentry-side outage, not a build problem. Look at Sentry's status page. If Sentry itself is degraded, this is expected — wait and re-run the workflow once Sentry recovers. Do not weaken the gate to work around a transient outage.
- Check
SENTRY_AUTH_TOKENvalidity. A 401/403 from the smoke-check step (visible in its script output) means the token is revoked/expired/missing — rotate it in Doppler/GitHub secrets, it is the sameSENTRY_AUTH_TOKENused by every other Sentry step in this workflow (dSYM upload, JS source-map upload). No new secret exists for this gate. - Check
SENTRY_ORG/SENTRY_PROJECT. Both are hardcoded plainenv:values in the workflow (professional-performance-portf/ppp-mobile) — a 404 from the smoke-check means one of these no longer matches the real Sentry project slug (e.g. after an org rename). - If dSYMs are genuinely missing or stripped, check whether
ExportOptions.plist'sstripSwiftSymbolssetting or an Xcode build-settings change removed debug info before the archive step ran. The archive (not the exported IPA) is what step 1 uploads from — if the archive itself lacks dSYMs, something upstream in the build (a scheme/config change, aDEBUG_INFORMATION_FORMATregression) needs investigating. - Do not bypass this gate to unblock a release. If you believe a failure is a false positive in the gate itself (not a real symbolication problem), that is a bug in the gate — fix the gate (or the script) and re-run, rather than reverting to
continue-on-erroror skipping the step. Weakening this gate reopens exactly the #10649 blind spot it exists to close.
Symbolicating a crash report by hand (no Sentry access)
The gate above guarantees Sentry can symbolicate. It does not help you if you are holding a .ips pulled off someone's device and have no SENTRY_AUTH_TOKEN — and no token exists outside CI. That was the state of the world until #11000: the dSYMs existed only inside Sentry, the shipped IPA is stripped (~105 text symbols; ExportOptions.plist sets stripSwiftSymbols, and nearest-symbol offsets land hundreds of KB from the real frame), so atos against the binary alone produced nothing trustworthy. 30 auto-filed crash issues, 0 with a usable stack frame.
Every build now retains its symbols as a CI artifact. This is additive — Sentry stays the primary automatic path; the artifact is the manual path.
| Platform | Artifact | Contains | Retention |
|---|---|---|---|
| iOS | ios-dsyms-<profile> | Every .dSYM from the archive (app and frameworks — Hermes, React), plus dsym-uuids.txt and README.txt | 30 days |
| Android | android-symbols-<profile> | native/ unstripped .so libraries, native-build-ids.txt, mapping/ R8 mapping files when minification is enabled | 30 days |
iOS — from a .ips to a named frame
Find the build. The crash report's
Binary Imagessection lists each loaded binary with its UUID and load address. Match the app's UUID against builds in the Actions tab — or just find themobile-build-native.ymlrun whose version/build number matches the report header.Download
ios-dsyms-<profile>from that run's artifacts (Actions → the run → Artifacts). No Sentry access needed; repo read access is enough.Confirm the UUID.
dsym-uuids.txtin the artifact maps every UUID to its.dSYMbundle. If the report's UUID is not in that file, you have the wrong build — go back to step 1 rather than symbolicating against the wrong symbols.Symbolicate the frame:
bashatos -o <Name>.dSYM/Contents/Resources/DWARF/<Name> \ -arch arm64 \ -l <load address from Binary Images> \ <frame address>Frames inside
hermes/Reactcome from those frameworks' own dSYM bundles, not the app's — pick the bundle whose UUID matches that frame's binary inBinary Images.
CI proves the artifact matches the shipped binary. The Verify dSYM UUIDs match the shipped IPA and stage for retention (#11000) step unzips the exported IPA, reads its main executable's LC_UUIDs with dwarfdump -u, and asserts every one is present among the archived dSYMs — failing the job (before the TestFlight submit) if any is missing. An artifact that does not match the shipped binary is worse than no artifact, because it looks like coverage.
Android — from a native crash to a named frame
Android release builds are not minified (no expo-build-properties plugin overrides Expo's android.enableProguardInReleaseBuilds=false default), so Java/Kotlin frames are already readable and mapping/ is normally empty. That is expected, not a fault. The files that matter are the unstripped native libraries in native/ — the Android analogue of dSYMs, and the ones that carry Hermes.
- Download
android-symbols-<profile>from the matching workflow run. - Match the crash frame's library by GNU build ID using
native-build-ids.txt. llvm-addr2line -f -C -e native/<abi>/<lib>.so <offset>(orndk-stackif you have the full logcat trace).
The Android side is deliberately advisory, not a gate. Its collection step never fails the build: the merged_native_libs layout shifts between AGP versions and there is no #10831-equivalent Android gate for it to sit behind, so a hard assertion on that path would risk bricking Android CD to protect symbols the team does not currently have at all. If it finds nothing it emits a ::warning:: and a job-summary line rather than silently publishing an empty artifact. The build-ID cross-check against the shipped APK is a warning for the same reason. Only iOS carries the blocking UUID gate.
Verifying the gate is working
- A successful run's job summary shows three green checkmarks: dSYMs uploaded, dSYM verification passed (with a bundle count), and the symbolication smoke-check passed (with a UUID count).
- To confirm end-to-end, force a native crash on a TestFlight build built after this gate shipped and check that the resulting Sentry issue resolves a full native stack trace (function names, file/line), not raw addresses.
Related
mobile/pppCI/CD reference — fullmobile-build-native.ymlworkflow reference- #10589 — original dSYM upload (soft)
- #10816 — local dSYM validity check + Sentry release health (soft)
- #10831 — this gate (hard)
- #10828 — parent epic: making the JSI-crash class un-shippable
- #11000 — dSYM/symbol retention as a CI artifact (the manual, token-free route above)
- #10964 — the live Hermes SIGSEGV that #11000 unblocks triage for