Manual Testing

Overview

This document provides guidelines for conducting manual testing within the Appcircle platform. It outlines the testing environments, the release testing workflow, and the critical modules that require special attention during manual validation.

The goal of manual testing is to ensure that new developments, improvements, and bug fixes function as expected and do not introduce regressions into existing features.

QA engineers should follow the environment guidelines and testing priorities described in this document when preparing and executing manual test plans.

Testing Environments

Appcircle maintains multiple environments for development, validation, and production usage. Each environment has a specific purpose and testing policy.

Dev Environment

The Dev environment is primarily used by developers to push their changes and perform initial testing.

Developers are expected to validate their work in this environment before it becomes available for broader testing.

QA engineers may also use the Dev environment for: - Early-stage validation of important improvements - Initial verification of bug fixes - Testing features that require early QA feedback

Since this environment frequently changes, stability is not guaranteed.

Prep Environment

The Prep environment is identical to the Production environment and serves as the primary environment for release testing.

All release candidates must be validated in this environment before they can be deployed to production.

Key principles: - Release testing is mainly conducted in this environment. - Issues detected here are considered high risk, as they may also exist in production. - Deployments to the Production environment must not occur without successful validation in Prep.

Production Environment (Prod)

The Production environment is the live environment used by Appcircle cloud customers.

Testing activities in this environment must be minimal and controlled.

QA engineers should perform only critical validation tests after deployments to confirm that the release has been successfully applied.

Guidelines for production testing: - Only perform essential verification tests after releases. - Bugs discovered in production should be immediately reported to the development team. - Environment-wide or destructive actions must not be executed without authorization.

Example of restricted operations: - Retention Period triggers - Environment-wide cleanup operations

Self-Hosted Environment (my.self)

The Self-Hosted environment is used to validate releases intended for customers who run Appcircle on their own infrastructure.

This environment is used by the QA team to ensure compatibility with self-hosted deployments.

Testing in this environment is important because: - Some behaviors may differ between Cloud and Self-Hosted environments - Configuration differences may reveal issues not seen in cloud deployments

All releases should be re-tested in this environment before being packaged and delivered to self-hosted customers.

Release Testing Process

The release testing workflow ensures that all developments included in a release are properly validated before deployment.

1. Release Scope Definition

The Development Management team determines the release scope and assigns the corresponding release version labels to tickets in Linear.

These labels allow the QA team to identify which tickets should be included in the manual test plan.

1. Test Planning

The QA team prepares a list of tickets included in the release and estimates the required testing effort.

This step helps determine: - Whether the release scope is feasible - Which areas require deeper validation - Which modules may be affected by the changes

1. Release Testing

Most manual tests are executed in the Prep environment.

During testing: - QA engineers validate each ticket based on its acceptance criteria. - Test results are tracked in Linear. - Tickets are marked as passed or failed based on the testing outcome.

1. Issue Evaluation

Failed tickets are reviewed by Product and Development leads to determine their severity.

The evaluation determines whether the issue is a release blocker.

Possible outcomes: - Fix required before release - Issue accepted and postponed to a future release

1. Release Approval

If all developments are tested and no blocking issues remain, the release proceeds to deployment.

1. Post-Release Verification

After deployment, critical scenarios must be re-tested in the Production environment to ensure the deployment completed successfully.

These tests focus on confirming that core workflows remain operational.

Manual Testing Focus Areas

Certain modules require additional attention during manual testing, especially when changes are made within those areas.

The following sections outline the most critical validation scenarios.

Build

Repository Provider Connection

Supported providers: - Azure - Bitbucket - GitLab - GitHub

What to test - Ensure build profiles can connect successfully to the repository providers. - Verify that previously connected repositories remain functional.

Connection Types

Supported connection methods: - Public - SSH - Personal Access Token (PAT) - OAuth

What to test - Verify that each connection type can be created successfully. - Ensure no authentication or permission errors occur.

Webhook Creation

Supported methods: - Manual webhook creation - Automatic webhook creation

What to test - Confirm that webhooks are created successfully on repository providers. - Verify that webhook-triggered builds work as expected.

Testing Distribution

Testing Portal Authentication

Supported authentication types: - None - Static - LDAP - SSO

What to test - Verify that each authentication type works correctly. - Ensure users can access the Testing Portal without authentication errors.

Binary Upload

Supported binary formats: - IPA - APK - AAB

What to test - Confirm that binaries can be uploaded successfully to Testing Distribution profiles.

Share with Testers

What to test - Verify that Android and iOS binaries can be shared with testers. - Confirm that sharing links are generated correctly.

Binary Re-sign

Supported methods: - Manual re-sign - Automatic re-sign

What to test - Ensure both re-sign mechanisms function correctly.

Binary Download and Installation

What to test - Verify that binaries can be downloaded from the Testing Portal. - Confirm that installations work correctly on supported mobile devices.

Enterprise App Store

Portal Authentication

Supported authentication types: - None - Static - LDAP - SSO

What to test - Verify that authentication methods function correctly for the Enterprise Portal.

Single Session

What to test - When disabled, ensure users are not forced out of active sessions. - When enabled, ensure users cannot maintain multiple active sessions.

Binary Channel Publishing

Supported channels: - Live - Beta - None

What to test

Validate the following transitions: - None → Beta - None → Live - Beta → Live - Live → None - Beta → None

In-App Update

What to test - Ensure Enterprise App Store profiles can generate valid In-App Update secrets.

Binary Download and Installation

What to test - Verify that binaries are visible in the Enterprise Portal. - Confirm that users can download and install the binaries.

Publish to Stores

Store Connections

Supported stores: - App Store - Huawei AppGallery - Google Play - Microsoft Intune

What to test - Verify that publish profiles can successfully connect to the stores.

Publish Flow Management

What to test - Confirm that publish flows can be edited. - Verify that publish flow lists load correctly.

Mark as Release Candidate (RC)

What to test - Ensure binaries with valid bundle IDs or package IDs can be marked as RC. - Verify that RC status can be removed correctly.

Binary Re-sign

What to test - Verify that both manual and automatic re-sign processes function correctly.

Auto Publish

What to test - Ensure binaries automatically trigger the publish flow when the feature is enabled.

Organization

License Count

What to test

Ensure license usage counts increase correctly when the following modules are used: - Build - Testing Distribution - CodePush - Publish to Stores - Enterprise App Store

Retention Period

What to test - Ensure artifacts are not deleted when the module retention setting is configured as Never.