Mobile App Cold-Start Testing: Launch Time & Failures

Mobile app cold-start testing verifies how an app launches when its process is not running, how long users wait for its first frame and usable initial screen, and whether initialization can recover from failures. A repeatable test establishes the application-data, account, permission, and entry-point conditions before measuring. It also tests onboarding, permission denial, interrupted setup, and relaunch after failure. The core model is launch/process state x application-data state x permission state, with the launch entry point and account state recorded alongside it. A clean-install onboarding flow and a returning user’s cold start perform different work and need separate baselines.

Use this five-step workflow:

• Choose and verify the startup state and entry route.
• Define first-frame, rendered-ready, and first-run milestones.
• Prepare a representative device and repeatable fixture.
• Repeat native measurements and exercise failure paths.
• Inspect timings, failed starts, and traces against a declared release gate.

Android provides cold, warm, and hot startup definitions plus TTID and TTFD measurements. On iOS, process prewarming and cached system state affect launch conditions; describe what the test controlled rather than claiming every launch had identical system conditions.

Table of Content

• 1. Cold Start, First Run, and Resume Are Different Test States
• 2. Define What “Launch Complete” Means Before You Measure It
• 3. Build a Repeatable Cold-Start Benchmark Setup
• 4. Measure Android Cold Start with Macrobenchmark
• 5. Measure iOS Launch Time with XCTest and Xcode Tools
• 6. Test Initialization Failures, Not Just Slow Initialization
• 7. Validate Permissions as Part of the First-Run State Machine
• 8. Test the First-Run Experience End to End
• 9. Turn Startup Results into Release Gates and Production Monitoring
• 10. Cold-Start Test Matrix and Evidence Checklist
• Conclusion

1. Cold Start, First Run, and Resume Are Different Test States

Before designing a test, define exactly what state the application is in.

Process state: cold, warm, hot, and resumed

On Android, StartupMode.COLD means the application process isn’t alive and must be created as part of startup. A warm start recreates an activity while the process remains alive, while a hot start brings an existing activity back to the foreground.

That distinction matters because a cold start includes operating-system and application initialization work that a hot start can avoid.

iOS should not be mapped mechanically onto the same three labels. Apple can prewarm an application by creating its process and loading linked libraries before any application code executes, then suspending that process until a later launch. The system can perform this after reboot and at other times depending on system conditions.

For test reporting, record the condition you actually controlled instead of claiming a stronger state than you can prove. For example, use descriptions such as “application not running before launch on physical iPhone” rather than asserting that every run represents an identical hardware-level cold state.

Installation state: clean install versus returning user

A cold start does not imply a clean installation. Compare these five startup scenarios:

• Test state: Returning-user cold start | Process: Not running | Application data: Existing | Likely behavior: Load stored account/configuration and main screen
• Test state: Verified first-run launch | Process: Not running | Application data: Verified first-run fixture | Likely behavior: One-time setup, onboarding, consent, permission decisions
• Test state: Restored-install launch | Process: Not running | Application data: Backup-restored state | Likely behavior: Validate restored data, account handling, and the intended onboarding path
• Test state: Returning-user resume | Process: Running/suspended | Application data: Existing | Likely behavior: Resume an existing session
• Test state: Interrupted first run | Process: Not running or restarted | Application data: Partially initialized | Likely behavior: Recover onboarding/setup state

The first three rows may all involve a new process, while their initialization workloads and expected screens differ.

If QA clears application data before every benchmark iteration, database creation, software development kit (SDK) setup, content extraction, onboarding, or configuration downloads may become part of every measurement. Use that workload for first-run performance and a verified existing-data fixture for a returning user’s cold start.

Define clean as an observed fixture, not simply an uninstall/reinstall action. Android Auto Backup can restore app data after installation and before the first user launch. Verify onboarding flags, local schema/data revision, account/session state, and authorization after preparation; record restore policy and test a restored installation separately.

Reset remote test-account and backend records independently when the scenario requires it. Clearing local app data does not create a new remote account. On either platform, record what was reset and read back the state the test depends on.

For upgrade-specific migration, retained permissions, sessions, and routing, use Codoid’s mobile app update testing guide. A verified first-run fixture does not replace an upgrade fixture.

Permission state is a separate variable

Permissions introduce another state dimension. A clean installation may begin with a permission in a never-requested state. A returning user may have granted it, denied it, revoked it later in Settings, or reached another platform-specific authorization state.

Android recommends requesting runtime permissions in context and allowing an application to continue gracefully if access is denied or revoked. Apple similarly treats access to protected resources as permission that the user can grant or deny.

Therefore, do not define a test simply as: Cold start, permission test. Define it more precisely:

• Launch state: Cold | Data state: Clean | Permission state: Not determined | Expected path: First-run flow; request only when required
• Launch state: Cold | Data state: Existing | Permission state: Granted | Expected path: Feature loads without another request
• Launch state: Cold | Data state: Existing | Permission state: Denied | Expected path: App remains usable; restricted feature explains recovery
• Launch state: Cold | Data state: Existing | Permission state: Revoked | Expected path: State is rechecked rather than assumed
• Launch state: Cold | Data state: Partial first run | Permission state: Mixed | Expected path: Flow resumes without duplicating completed steps

2. Define What “Launch Complete” Means Before You Measure It

A fast splash-screen transition does not necessarily mean the application is ready. QA should normally track at least two milestones:

First visible frame: When the app first draws its interface; essential content or interactions may still be unavailable.

Application ready: When required initial content is rendered and the primary action is usable. Define a matching functional assertion and an application-owned signal for this condition.

(Figure 1 description: Android measurement boundaries, not a captured trace. Flow goes System receives launch intent -> First frame (may be a loading shell) -> App reports fully drawn -> Containing/following frame completes. TTID covers the first frame. TTFD covers the fully drawn frame. iOS metrics use their own documented boundaries.)

First frame: TTID or platform launch duration

Android calls its first-frame startup metric time to initial display (TTID). StartupTimingMetric measures the duration from the system receiving the launch intent to the first frame of the destination activity.

On iOS, Xcode Organizer’s launch metric measures to the application’s first screen after the static launch screen. XCTest exposes separate launch metrics with configurable behavior; record the initializer used instead of treating all XCTest measurements as first-frame-only.

First-frame measurement answers: “How long did the user wait before the app showed its initial interface?” It does not necessarily answer: “Could the user actually use it?”

Fully displayed or interactive-ready milestone

Android provides time to full display (TTFD) using the app’s fully-drawn signal; StartupTimingMetric can report timeToFullDisplayMs. Validate the signal against the agreed rendered-ready condition. A number alone does not prove that the app reported readiness at the correct point.

For View-based Android applications, this can involve reportFullyDrawn(). For Compose applications, Android recommends the Compose-aware ReportDrawn, ReportDrawnWhen, or ReportDrawnAfter utilities instead of manually calling the activity method.

Do not define “fully drawn” as “every startup request in the application has finished.” Define the smallest product-relevant condition that makes the initial screen genuinely usable.

For example, an application’s readiness contract might be:

• App: News app | Reasonable ready condition: Initial feed rendered and scrolling available
• App: Banking app | Reasonable ready condition: Authenticated landing state resolved and primary actions enabled
• App: Messaging app | Reasonable ready condition: Conversation list available; background attachment sync may continue
• App: Retail app | Reasonable ready condition: Main navigation and initial catalog state usable

These are illustrative definitions, not platform requirements.

On iOS, keep the selected XCTest launch metric, app-defined intervals, and functional readiness assertions distinct. XCTApplicationLaunchMetric(waitUntilResponsive: true) waits for main-thread responsiveness after the first frame; OSSignposter measures work between app-defined begin/end events. Neither automatically proves that required business data is rendered.

First-run completion is a different metric

Suppose the application renders an onboarding screen after 700 ms, then the user reads three screens and decides whether to allow location access. The launch did not take 20 seconds just because the human onboarding interaction lasted that long.

Track separately:

• Launch duration: System launch to first frame or readiness milestone.
• First-run journey duration: Launch through required setup/onboarding completion.

This separation makes performance regressions diagnosable while still allowing QA to evaluate whether the complete first-run experience works.

3. Build a Repeatable Cold-Start Benchmark Setup

A fixture is the prepared test state: local data, account/session, onboarding flags, and permissions. Preparation must be repeatable, and verification must fail visibly when those inputs differ. Keep login, seeding, resets, and human permission decisions outside the measured launch.

• Variable: Device and OS | Record before comparison: Exact model, operating-system version, intended Android user where applicable
• Variable: Build | Record before comparison: App identifier, build/commit, release-like variant, toolchain and resolved dependencies
• Variable: Fixture | Record before comparison: Local schema/data revision, onboarding state, account/session, restore policy
• Variable: Permissions | Record before comparison: Actual authorization, relevant permission flags, app-owned prompt-history state
• Variable: Entry point | Record before comparison: Launcher, notification, deep link, or universal link; expected destination
• Variable: Network and backend | Record before comparison: Connection profile, application programming interface (API) environment, account data and response policy
• Variable: Device condition | Record before comparison: Unlocked screen, thermal/battery state and relevant background activity
• Variable: Benchmark policy | Record before comparison: Compilation/profile mode, iteration count, readiness deadline and valid-trial rules

Use representative devices and release-like builds

Use physical devices for performance baselines and release gates. Emulators and simulators remain useful for functional state, permission, and recovery coverage. Android’s Macrobenchmark guide flags configurations such as emulators, debuggable targets, and low battery; investigate those conditions instead of suppressing warnings to obtain a number. Apple’s performance guidance likewise calls for release configuration and user-like conditions.

Compare equivalent device/OS/fixture combinations. Include a slower supported device when that represents the audience. A profile change or a different account dataset can change the measured workload even when the app build looks similar.

Repeat measurements and account for every attempt

Use a declared iteration count and retain the individual samples. The 10 Android iterations and five iOS iterations below illustrate API configuration; they are not validated statistical sample sizes. Median comparisons are useful for controlled regression detection, while production P50 (median) and P90 (90th percentile) views describe a broader field population. Small laboratory batches do not establish a stable production tail.

Count crashes, startup hangs, and readiness timeouts separately from successful durations. Record invalid fixture or capture trials with their reasons. The release-gate example explains how these outcomes affect the decision.

4. Measure Android Cold Start with Macrobenchmark

Macrobenchmark collects native startup metrics and traces. Example environment reviewed against primary documentation on October 12, 2026: AndroidX Benchmark 1.5.0 and the released dependencies below. These versions identify the example; they are not an executed test record.

Configure StartupMode.COLD and StartupTimingMetric

Create a separate com.android.test module using Android Studio’s Benchmark module wizard. Set targetProjectPath = “app” and the runner to androidx.test.runner.AndroidJUnitRunner; select matching benchmark variants. In this standalone test module, place benchmark code under src/main and declare its dependencies with implementation. Keep Google Maven available.

// macrobenchmark/build.gradle.kts: dependency fragment
dependencies {
    implementation("androidx.benchmark:benchmark-macro-junit4:1.5.0")
    implementation("androidx.test.ext:junit:1.3.0")
    implementation("androidx.test:runner:1.7.0")
    implementation("androidx.test.uiautomator:uiautomator:2.4.0")
}
  

These released dependency versions provide the APIs used below; record the resolved dependency graph and project toolchain when validating the integration.

For the target app, copy the release build type into a locally signed benchmark variant, set isDebuggable = false, and preserve release minification/resource-shrinking behavior. Include <profileable android:shell="true" /> inside <application> in its benchmark manifest. Add implementation("androidx.profileinstaller:profileinstaller:1.4.1") to the target. ProfileInstall supports profile/capture operations; installing that library does not create the app’s Baseline Profile.

Uncompiled integration template, application adapters required. This abstract class requires a concrete subclass with the application ID and fixture implementation. The readiness resource is an illustrative application-owned acknowledgment, not an Android API.

import androidx.benchmark.macro.BaselineProfileMode
import androidx.benchmark.macro.CompilationMode
import androidx.benchmark.macro.MacrobenchmarkScope
import androidx.benchmark.macro.StartupMode
import androidx.benchmark.macro.StartupTimingMetric
import androidx.benchmark.macro.junit4.MacrobenchmarkRule
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.filters.LargeTest
import androidx.test.filters.SdkSuppress
import androidx.test.uiautomator.By
import androidx.test.uiautomator.Until
import org.junit.Assert.assertTrue
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith

@RunWith(AndroidJUnit4::class)
@LargeTest
abstract class ReturningUserColdStartBenchmark {
    @get:Rule val benchmark = MacrobenchmarkRule()

    protected abstract val targetPackage: String

    protected abstract fun seedAndVerifyReturningUser(scope: MacrobenchmarkScope)

    protected abstract fun assertInitialUiReady(scope: MacrobenchmarkScope)

    @Test
    @SdkSuppress(minSdkVersion = 34)
    fun launcherColdStart() = benchmark.measureRepeated(
        packageName = targetPackage,
        metrics = listOf(StartupTimingMetric()),
        compilationMode = CompilationMode.Partial(
            baselineProfileMode = BaselineProfileMode.Require
        ),
        startupMode = StartupMode.COLD,
        iterations = 10,
        setupBlock = {
            seedAndVerifyReturningUser(this)
            pressHome()
        }
    ) {
        startActivityAndWait()
        assertTrue(
            "Ready/fully-drawn acknowledgment timed out",
            device.wait(
                Until.hasObject(
                    By.res(targetPackage, "startup_ready_after_draw")
                        .enabled(true)
                ),
                10_000L
            )
        )
        assertInitialUiReady(this)
    }
}
  

Fixture contract. seedAndVerifyReturningUser must establish onboarding completion, the agreed data/schema revision, account/session state, and actual authorization, then read them back and assert the result. It may leave data untouched only after verifying a match. Persist writes, record the fixture revision and iteration without secrets, and fail setup on mismatch. Use UI flows or hooks confined to the test build; an empty helper is not a fixture. Compare first- and final-trial state evidence so repeated setup cannot conceal drift.

Preparation belongs in setupBlock. With StartupMode.COLD, Macrobenchmark stops the target process between setup and measurement. Neither login nor pm clear belongs in the measured returning-user launch. This sample requires a physical API 34+ device because compilation reset can reinstall the APK on earlier Android versions. For older devices, document a separately managed compilation procedure and use CompilationMode.Ignore() with its experimental API opt-in, or rebuild and verify the fixture after compilation’s reinstall.

The example deliberately requires a packaged Baseline Profile. BaselineProfileMode.Require fails when the expected profile or ProfileInstaller is absent. DEFAULT can continue without a profile; Ignore() leaves compilation unchanged. Choose and record the intended policy instead of silently changing modes to make a run pass.

Run the concrete subclass through ./gradlew :macrobenchmark:connectedCheck. Record Android Gradle Plugin (AGP), Gradle, Java Development Kit (JDK), compile SDK, app build, and device/OS. The ten-second acknowledgment wait starts after startActivityAndWait() returns; it is not the total launch-to-ready budget. Enforce that budget through native timings and a separate functional deadline. Assertion failure may stop the batch early, so record actual launches started. Ten iterations and ten seconds are illustrative settings.

Capture TTID and TTFD

Inspect timeToInitialDisplayMs for the initial frame and timeToFullDisplayMs for app-signaled full display. Macrobenchmark’s TTFD boundary includes completion of the frame containing or following the fully-drawn signal. A first frame may still be a loading shell.

Readiness acknowledgment contract. Reset startup_ready_after_draw on every launch. Expose it only after required content and primary actions are rendered, fully-drawn reporting occurs, and its containing/following frame completes. A “network finished” flag or a marker inserted before drawing is insufficient. Compose apps can expose an equivalent tag through testTagAsResourced; use a selector matching their actual semantics. The UI wait keeps capture open; its elapsed time is not TTID or TTFD.

Implement assertInitialUiReady as an independent UI assertion of the intended destination, required visible content, and enabled primary action. Reading the same acknowledgment again does not verify that contract.

For Compose, connect the real readiness condition to ReportDrawnWhen, ReportDrawnAfter, or the appropriate reporting utility. For Views, place reportFullyDrawn() at the agreed usable boundary. The signal still needs a functional assertion.

If expected TTFD is missing, check:

• Whether the correct launch activity emits the signal.
• Whether the trace includes its containing/following frame.
• API/tool support and capture errors.
• Whether the readiness condition is reached and correctly defined.

Classify a capture/instrumentation defect as an invalid timing trial; classify a genuine crash, hang, or readiness timeout as a failed start. Neither may pass unnoticed. Do not replace absent TTFD with the UI-wait duration.

Open Perfetto when a regression appears

Retain JSON output and per-iteration traces. Examine expensive initialization, main-thread work, I/O, library loading, and contention to connect the timing regression to a cause. Before treating this integration as validated, demonstrate that delaying readiness moves TTFD, that its frame remains captured, and that a timeout fails visibly. Attach the device results and trace references; documentation review alone does not provide those results.

5. Measure iOS Launch Time with XCTest and Xcode Tools

Use XCTest for repeatable launch measurements, Instruments for application-defined intervals, and Xcode Organizer for shipping-app evidence. Label each result with its measurement boundary; these tools answer related questions about startup rather than producing interchangeable numbers.

Configure the UI-test target and performance scheme

Create an iOS UI Testing Bundle in the application’s Xcode project, select the app under Target Application, and include the bundle in the scheme’s Test action. Use a released Xcode version that supports the physical test device. The examples below assume an iOS 15 or later deployment target, which covers the application-side OSSignposter and async URLSession APIs.

Set the Test action’s build configuration to Release, turn off Debug executable, and disable code coverage and runtime sanitizers in the performance test plan. Record the actual Xcode build, SDK, Swift language mode, device/OS, app build and fixture revision. Apple recommends this configuration to approximate the conditions in which production code runs.

Prepare and verify the selected account, local data, onboarding and authorization state before measurement. Keep fixture creation outside the timed startup path, and verify that repeated launches preserve the required state. A returning-user test must not silently become a first-run test.

Place this illustrative example in the configured UI-test target:

import XCTest

final class LaunchPerformanceTests: XCTestCase {
    @MainActor
    func testLaunchPerformance() {
        continueAfterFailure = false
        let app = XCUIApplication()
        let options = XCTMeasureOptions()
        options.iterationCount = 5

        measure(
            metrics: [
                XCTApplicationLaunchMetric(waitUntilResponsive: true)
            ],
            options: options
        ) {
            app.launch()
        }
    }
}
  

app.launch() terminates an existing instance before launching the application. This establishes a new process launch while retaining application data. With waitUntilResponsive: true, the metric tracks the first frame and subsequent main-thread responsiveness. It does not prove that required initial content has arrived or a business journey works.

Five iterations are an illustrative starting count; apply the release-gate stability policy when choosing the actual sample size. Pair the metric with a separate functional check of the expected content, enabled primary action and resulting destination. A bounded UI wait supports that assertion; its host-side elapsed time is not the native launch metric.

Signpost awaited application work with an explicit boundary

The following application-target function measures one initial HTTP data load and response validation. Its InitialDataLoad name describes that scope: decoding, UI updates and rendering are outside the interval. The begin event also excludes any OS or pre-main work that occurred earlier.

import Foundation
import os

@available(iOS 15.0, *)
func loadInitialData(
    from url: URL,
    using session: URLSession
) async throws -> Data {
    let signposter = OSSignposter(
        subsystem: "com.example.app",
        category: .pointsOfInterest
    )
    let state = signposter.beginInterval(
        "InitialDataLoad",
        id: signposter.makeSignpostID()
    )
    do {
        let (data, response) = try await session.data(from: url)
        guard let http = response as? HTTPURLResponse,
              (200...299).contains(http.statusCode) else {
            throw URLError(.badServerResponse)
        }
        signposter.endInterval(
            "InitialDataLoad", state, "outcome=success"
        )
        return data
    } catch {
        signposter.endInterval(
            "InitialDataLoad", state, "outcome=failure"
        )
        throw error
    }
}
  

Call and await this function from the actual initialization flow. Supply a session with the product’s explicit timeout and cache policy. The interval ends after success or a thrown failure, then returns data or propagates the error. Cancellation that throws follows the failure path. Process termination or work that never returns can leave an incomplete interval; treat that trial as interrupted or failed.

In Swift, defer runs when its enclosing scope exits. Launching detached work and immediately exiting that scope would measure scheduling, not the detached work’s completion. Keep the awaited operation inside the interval. Inspect its start, end and outcome in Instruments. If a separate metric is named initial screen ready, connect its end to the agreed rendered and usable condition rather than data completion alone.

Record launch conditions and compare field evidence

Report the condition controlled, such as repeated process launches on a physical iPhone with a returning-user fixture. Apple describes cold/warm launches as a spectrum affected by device state and also documents process prewarming. Record after-reboot or memory-pressure scenarios separately when they are relevant.

Use Organizer’s device filters and P50/P90 launch views to assess shipping releases. Its first-screen boundary remains distinct from the signposted data interval and product readiness. Retain physical-device XCTest results, Instruments evidence and a failure-path result before claiming execution validation. These examples were reviewed against primary documentation; they have not been compiled or run against a supplied application.

6. Test Initialization Failures, Not Just Slow Initialization

A fast startup that enters a broken state is not a successful startup. Mobile app cold-start testing should therefore combine performance measurement with failure injection.

Map the initialization dependency chain

Before writing tests, identify what truly has to happen before the initial screen is usable.

Potential startup dependencies include local database access, persisted configuration, authentication/session restoration, required remote configuration, migrations, cryptographic material, and SDKs needed for the first journey.

Android’s App Startup library explicitly supports defining initialization dependencies and lazily initializing components that do not need to run immediately. Android’s broader startup guidance likewise recommends deferring nonessential work until after startup.

That architectural inventory is also the starting point for QA’s failure model.

Inject dependency failures deliberately

An illustrative initialization test matrix might look like this:

• Preconditions: Returning user, valid session | Injected condition: Network unavailable | Expected outcome: App reaches usable offline/error state within bounded time | Assertions: No infinite spinner; retry available where appropriate | Evidence: Video, logs, trace, state
• Preconditions: Clean install | Injected condition: Configuration endpoint timeout | Expected outcome: First-run flow explains/retries failure safely | Assertions: No partially completed setup marked successful | Evidence: Network trace, app logs
• Preconditions: Returning user | Injected condition: Startup API returns 5xx | Expected outcome: Local navigation remains stable; recoverable state shown | Assertions: Relaunch/retry does not corrupt data | Evidence: HTTP evidence, screenshots
• Preconditions: Partial setup exists | Injected condition: App killed during initialization | Expected outcome: Next launch resumes or restarts safely | Assertions: No duplicate records or endless onboarding loop | Evidence: Before/after storage state
• Preconditions: Existing local data | Injected condition: Required local record unreadable | Expected outcome: Error is contained and recovery policy executes | Assertions: No crash loop; data handling matches product policy | Evidence: Crash/log evidence
• Preconditions: Slow dependency | Injected condition: Response exceeds agreed deadline | Expected outcome: UI remains responsive and timeout policy applies | Assertions: No unbounded blocking on startup | Evidence: Trace and timing
• Preconditions: Test-only fault enabled | Injected condition: Main thread stalls during launch | Expected outcome: Readiness deadline fails; capture OS termination if it occurs | Assertions: Classify as a failed start; verify next-launch recovery | Evidence: Main-thread trace; ANR/watchdog report if generated
• Preconditions: Required async work pending | Injected condition: Completion never arrives | Expected outcome: Bounded timeout or documented recovery state | Assertions: No indefinite successful loading state | Evidence: Readiness failure and incomplete interval
• Preconditions: Initialization in progress | Injected condition: Process terminated before completion | Expected outcome: Interrupted startup; subsequent launch recovers | Assertions: Incomplete interval is not a successful duration | Evidence: Termination evidence, stored state, relaunch result

These scenarios are illustrative. The expected behavior depends on the product’s offline policy and whether the dependency is mandatory. An Android application-not-responding (ANR) event or an iOS watchdog termination needs OS diagnostic evidence; a harness timeout alone does not prove either occurred. Classify a launch that misses its readiness contract as failed even when no OS report is generated. Keep fault injection in an isolated test build or controlled test-only path, separate from ordinary benchmark batches.

An intentionally injected failed launch can still produce a passed recovery test when the specified containment and relaunch assertions succeed. Report these cases separately from nominal baseline/candidate launches. A handled dependency timeout may satisfy an agreed degraded-ready contract; it is not automatically a startup failure.

Verify retry, idempotency, and recovery

Startup logic frequently contains “run once” operations. Those operations become dangerous when failure occurs between: work performed and completion recorded.

Test what happens if the application is terminated at that boundary and then relaunched. The second attempt should not accidentally create duplicate database rows, submit duplicate registration calls, repeatedly migrate the same data, or leave a persistent state that can never finish initialization.

Evidence should also be useful to engineering. Capture a stable initialization stage or trace event, failure category, duration, retry count, and resulting state, but avoid logging tokens, secrets, or sensitive user data.

7. Validate Permissions as Part of the First-Run State Machine

Treat permission decisions as persisted state changes. Request protected access in context, verify current authorization before use, and keep unrelated navigation available after denial.

Cover the states each permission actually supports

• State: Never requested | What QA should verify: Request appears at the relevant feature; app-owned prompt history matches the fixture
• State: Granted | What QA should verify: Feature works without redundant prompts
• State: Denied or dialog dismissed | What QA should verify: Distinguish the resulting OS state; restricted functionality explains recovery
• State: Revoked in Settings | What QA should verify: Authorization is rechecked; relaunch recovers if the OS terminated the process
• State: One-time grant, where supported | What QA should verify: Exercise use, background/expiry or revocation, then relaunch and recheck
• State: Restricted or mixed authorization | What QA should verify: The UI respects OS restrictions and preserves unrelated permission state

Android’s one-time grants apply to supported camera, microphone, and location permissions, not notification permission. Their lifetime depends on platform behavior and use; verify the actual result rather than waiting an invented universal expiry interval.

Reset Android notification permission and verify the result

This Android Debug Bridge (ADB) recipe follows Android’s Android 13+ notification test guidance. Use a dedicated test device and an installed app that declares POST_NOTIFICATIONS; a target SDK of 33 or later lets the app control request timing. Replace the example package with the installed application ID. These commands affect that permission, not the local-data or backend fixture.

APP_PACKAGE=com.example.app
adb shell am force-stop "$APP_PACKAGE"
adb shell pm revoke "$APP_PACKAGE" \
  android.permission.POST_NOTIFICATIONS
adb shell pm clear-permission-flags "$APP_PACKAGE" \
  android.permission.POST_NOTIFICATIONS user-set
adb shell pm clear-permission-flags "$APP_PACKAGE" \
  android.permission.POST_NOTIFICATIONS user-fixed
adb shell pm permission-flags "$APP_PACKAGE"
adb shell dumpsys package "$APP_PACKAGE"
  

Inspect the runtime-permissions section for the intended Android user: POST_NOTIFICATIONS should show granted=false with neither USER_SET nor USER_FIXED; other flags may remain. On a device with multiple users, record adb shell am get-current-user and use the intended user consistently in the permission commands. Also verify the app’s own prompt-history and onboarding flags.

Launch the app, reach the feature, and test grant, deny, dismissal, Settings revocation, and relaunch. Read authorization again after each transition. Runtime checks should include ContextCompat.checkSelfPermission(...) and NotificationManagerCompat.areNotificationsEnabled(); per-channel settings remain separate.

Establish iOS authorization independently

Use controlled test-device or Simulator privacy state and XCUIApplication.resetAuthorizationStatus(for:) for supported protected resources. Record the resource, OS version, reset method, and observed authorization value. Reset controls are resource-specific; a reinstall is not evidence that every credential, account, and permission dependency has become clean. Recheck the state before the relevant feature uses it. Measure the first-run journey separately from launch readiness so time spent reading permission dialogs does not become the startup-performance number.

8. Test the First-Run Experience End to End

First-run testing should begin with a defined clean state and continue beyond “the onboarding screen appeared.”

Complete, skip, interrupt, and resume onboarding

Where the product supports these behaviors, test:

• Scenario: Complete onboarding | Assertion: Completion persists across relaunch
• Scenario: Skip optional onboarding | Assertion: Main journey remains available
• Scenario: Kill app midway | Assertion: Relaunch enters a valid resumable or restart state
• Scenario: Background/foreground midway | Assertion: State does not jump forward or reset unexpectedly
• Scenario: Relaunch after completion | Assertion: One-time screens don’t return without a valid reason

For every scenario, capture both visible UI state and persistent state where practical.

Start first run with network or backend failures

Apply the initialization-failure matrix to the verified first-run fixture. Confirm that setup can recover without reinstalling, completed work is not repeated, and a later relaunch recognizes the actual completion state. Include the backup-restored fixture as a separate case instead of assuming it follows the same onboarding route.

Combine onboarding with permission grant and denial paths

Do not test onboarding and permissions as independent scripts if they interact in production. A first-run flow may need to survive combinations such as:

• onboarding completed, permission denied, app killed, relaunch, feature requested again
• permission granted, network initialization fails, app relaunched, permission remains granted, setup resumes

State-combination testing finds failures that single-path scripts usually miss.

9. Turn Startup Results into Release Gates and Production Monitoring

Choose budgets from the app’s historical results, representative devices, user impact, and observed measurement variation. A slower app still has meaningful regressions: assess a 150 ms increase against its baseline and variation while addressing broader architectural work where needed.

Use a declared regression and rerun policy

Illustrative policy and values; these are not measured app results or universal targets. Before starting, choose the fixture/device combination, sample count, absolute/relative budgets, readiness deadline, and same-build noise tolerance. Estimate noise from repeated baseline batches on the same setup; select enough iterations to support the decision.

• Gate input: Started launches per build | Illustrative value: 10; count actual attempts, not just successful durations
• Gate input: Baseline / candidate median TTID | Illustrative value: 600 ms / 675 ms
• Gate input: Absolute / relative increase | Illustrative value: 75 ms / 12.5%
• Gate input: Investigation budget | Illustrative value: Increase exceeds both 50 ms and 10% of the baseline median
• Gate input: Same-build noise check | Illustrative value: Predeclared median-difference tolerance of 20 ms
• Gate input: Confirmation | Illustrative value: At most one additional matched baseline/candidate batch
• Gate input: Reliability gate | Illustrative value: No crashes, hangs, or missed readiness deadlines in nominal baseline/candidate launches

Calculate the absolute increase as candidate median – baseline median and the relative increase as absolute increase / baseline median x 100. Here, 75 ms exceeds 50 ms and 12.5% exceeds 10%, so the initial result triggers investigation. First verify the fixture and capture, then apply the single confirmation batch. Block the performance gate only when the regression is confirmed under stable comparable conditions; conflicting results or excessive noise are inconclusive and need diagnosis, not unlimited reruns until green.

Apply a separately chosen budget to the app’s valid TTFD or rendered-ready measurement. Improvement in TTID cannot compensate for broken readiness.

Keep failures and invalid measurements visible

Record every attempted launch. A crash, hang, or readiness timeout is a failed start. A successful UI journey with missing expected TTFD is an invalid timing trial, not proof of a fast launch. A fixture failure before launch is a preflight failure; list it separately because no launch was attempted. If a harness interruption leaves the launch outcome unknown, retain that unknown outcome and withhold a pass.

For a batch with 10 known launch outcomes and one readiness failure, the launch-failure rate is 1 / 10 = 10%. Report the timing summary with its valid sample count; nine successful timings do not erase the failed attempt. State the failure denominator, metric-invalid count, and unknown-outcome count next to the median. This illustrative zero-failure acceptance policy does not estimate production failure risk from only 10 trials.

Distinguish optimization goals from Android vitals

Android performance guidance suggests aiming for a cold start of 500 ms or less. Android vitals classifies a cold startup with TTID of 5 seconds or longer as excessive. These refer to a performance goal and a production-health classification respectively; the vitals threshold is not TTFD or onboarding duration.

Use those figures as context for an application-specific gate. A 4.9-second TTID can still be a poor user experience, and a credible improvement plan needs representative baselines rather than a copied universal pass/fail number.

Retain evidence and follow the release into production

Keep the candidate and baseline build IDs, exact fixture/device/entry route, compilation policy, every trial outcome, raw metrics, and trace links. After rollout, inspect field results by comparable device and OS populations. Production percentiles and lab medians answer different questions.

Codoid’s mobile app launch checklist covers broader go/no-go evidence. Its mobile testing strategy guide extends the device, network, and risk-based coverage plan.

10. Cold-Start Test Matrix and Evidence Checklist

Use this copyable matrix as a starting point. Unless a row says otherwise, launch with the app process absent and record the platform-specific procedure. Keep launcher, notification, and link results in separate baseline groups. Replace illustrative ready conditions with the product’s actual contract.

• ID: CS-01 | Entry point and account: Launcher; signed in | Data and permission fixture: Existing data; granted | Expected result: Main screen reaches rendered-ready baseline | Evidence: Native metric and trace
• ID: CS-02 | Entry point and account: Launcher; new account | Data and permission fixture: Verified first run; never requested | Expected result: First-run UI works; prompts are contextual | Evidence: State checks and UI record
• ID: CS-03 | Entry point and account: Launcher; signed in | Data and permission fixture: Existing data; denied | Expected result: Usable navigation and clear restricted-feature recovery | Evidence: Authorization and UI
• ID: CS-04 | Entry point and account: Launcher; signed in | Data and permission fixture: Existing data; revoked | Expected result: Authorization rechecked; safe recovery after relaunch | Evidence: State transition and logs
• ID: CS-05 | Entry point and account: Launcher; signed in | Data and permission fixture: Existing data; offline | Expected result: Bounded offline/error state; no endless spinner | Evidence: Network record and trace
• ID: CS-06 | Entry point and account: Launcher; signed in | Data and permission fixture: Existing data; API timeout | Expected result: Documented timeout/retry policy | Evidence: HTTP timing, recovery state, retry result
• ID: CS-07 | Entry point and account: Launcher; setup incomplete | Data and permission fixture: Partial first run; mixed grants | Expected result: Completed steps retained; flow resumes safely | Evidence: Stored state and UI
• ID: CS-08 | Entry point and account: Launcher; signed in | Data and permission fixture: Partial setup; previous kill | Expected result: Setup remains idempotent on retry | Evidence: Data comparison and logs
• ID: CS-09 | Entry point and account: Launcher; signed in | Data and permission fixture: Baseline fixture; slower device | Expected result: Comparable timing and ready-state assertion | Evidence: Samples and device metadata
• ID: CS-10 | Entry point and account: Relaunch; prior account | Data and permission fixture: Service restored; state retained | Expected result: Recovery reaches usable state without reinstall | Evidence: Retry and relaunch record
• ID: CS-11 | Entry point and account: Launcher; restored account | Data and permission fixture: Backup-restored installation | Expected result: Restored data verified; correct account/onboarding path | Evidence: Restore policy and markers
• ID: CS-12 | Entry point and account: Notification; signed in | Data and permission fixture: Existing data; needed grants | Expected result: Payload routes to intended usable destination once | Evidence: Payload ID, route, trace
• ID: CS-13 | Entry point and account: Deep/universal link; signed out | Data and permission fixture: Existing or first-run fixture | Expected result: Safe authentication; destination deferred and restored | Evidence: Link, auth state, route
• ID: CS-14 | Entry point and account: Launcher; test account | Data and permission fixture: Controlled stall or missing completion | Expected result: Failed-start classification; bounded recovery | Evidence: Timeout; OS report if present

For notification and link cases, verify the expected destination, authentication policy, deferred routing after sign-in, duplicate delivery handling where relevant, and the destination’s ready milestone. A fast first frame on the wrong screen fails the functional test.

Copyable evidence record

The fields below are an unfilled template, not a captured benchmark. Duplicate it per case or run and replace the instructions with real values.

• Field: Case, owner, and status | Value to record: Case ID; accountable owner; pass/fail/invalid/pending
• Field: Build and environment | Value to record: App/commit; toolchain/dependencies; device/OS; entry point
• Field: Fixture | Value to record: Revision; local/account/onboarding/permission state; restore/reset boundary
• Field: Ready contract | Value to record: Required content/action; signal; rendered-state assertion; deadline
• Field: Samples and outcomes | Value to record: Started, failed, unknown, metric-invalid and valid timing counts
• Field: Results | Value to record: Native TTID/TTFD or exact iOS metric; app interval; absolute/relative deltas
• Field: Evidence and decision | Value to record: Result/trace/log links; reproduction steps; gate outcome and approver

Retain real evidence for the ordinary path, delayed readiness, failed readiness, permission denial or revocation, and at least one supported non-launcher route. Include a screen-reader focus/recovery check, sensitive cached-content protection, or low-storage case when those risks apply to the product.

Conclusion

Reliable mobile app cold-start testing proves three things: the app began in the intended state, the timing covered the intended milestone, and initialization recovered according to the product’s policy. A quick first frame is useful, but it cannot compensate for missing content, incorrect routing, or a startup path that never recovers. Keep first-run and restored-install fixtures separate from returning-user benchmarks. Use native timings with a verified rendered-ready contract, retain traces, and count failures alongside successful samples. That evidence gives QA and engineering a repeatable basis for release decisions.

Explore Codoid’s mobile app testing services or our broader mobile test automation services for help validating startup performance and first-run reliability.

Ready to Optimize Your App's Launch?

Talk to a Mobile QA Expert

Frequently Asked Questions