API Automation Test Framework

Overview

The API Automation Test Framework is a comprehensive test automation suite for validating AppCircle's REST API endpoints across multiple environments. Built with Java, TestNG, and REST Assured, it provides extensive coverage for Build, Distribution, Publish, Signing Identity, Organization, and Enterprise App Store modules.

Note

This framework is actively maintained and used for continuous integration testing. All changes should be tested across relevant environments before merging.

Purpose

This framework serves as the primary quality assurance mechanism for AppCircle's API layer, ensuring:

• API Contract Validation: Verifies that API endpoints behave according to specifications
• Regression Testing: Prevents breaking changes from affecting existing functionality
• Multi-Environment Validation: Tests across dev, prep, prod, self-hosted, and self-hosted-k8s environments
• Automated Reporting: Generates detailed Allure reports for test execution results
• CI/CD Integration: Jenkins builds and publishes the Docker image used to run tests; GitHub Actions runs Checkstyle on pull requests

Architecture

Technology Stack

• Language: Java 11
• Test Framework: TestNG 7.8.0
• HTTP Client: REST Assured 5.3.0
• Reporting: Allure 2.23.0
• Build Tool: Maven (3.9.3 in the Docker image)
• Containerization: Docker
• CI/CD: Jenkins (image pipeline), GitHub Actions (Checkstyle on pull requests)

Project Structure

api-automation-test/
├── src/
│   ├── main/
│   │   └── resources/
│   │       ├── dev/
│   │       ├── prep/
│   │       ├── prod/
│   │       ├── self-hosted/
│   │       └── self-hosted-k8s/
│   └── test/
│       └── java/com/appcircle/apitest/
│           ├── modules/
│           │   ├── build/
│           │   ├── distribution/
│           │   ├── publish/
│           │   ├── signingidentity/
│           │   ├── organization/
│           │   ├── enterpriseStore/
│           │   ├── admin/
│           │   ├── dashboard/
│           │   ├── license/
│           │   ├── reservedEnvironment/
│           │   ├── swagger/
│           │   └── AbstractBaseTest.java
│           ├── annotations/
│           ├── listeners/
│           └── filters/
├── scripts/
├── testng.xml
├── pom.xml
└── Dockerfile

Base Test Class

All test classes extend AbstractBaseTest (com.appcircle.apitest.modules.AbstractBaseTest), which provides:

• Authentication: Automatic JWT token generation from PAT (Personal Access Token)
• Request Execution: Standardized GET, POST, PUT, DELETE methods
• Response Validation: Built-in status code validation and error handling
• cURL Generation: Automatic cURL command capture and Allure attachment
• Trace ID Logging: Captures trace IDs from response headers for debugging
• Test Skipping: Environment-based test skipping mechanism
• Timeout Configuration: Environment-specific timeout values for async operations

Key Features

Environment Support

The framework supports five distinct environments:

• dev: Development environment (default)
• prep: Pre-production/staging environment
• prod: Production environment
• self-hosted: Self-hosted AppCircle instances
• self-hosted-k8s: Kubernetes-based self-hosted instances

Each environment has its own configuration file located at src/main/resources/{environment}/config.properties.

Retry Mechanism

Tests can be marked with @RetrySafe annotation to enable automatic retry on failure:

• Retry Count: Currently set to 1 retry attempt (MAX_RETRY = 1)
• Scope: Can be applied at class or method level
• Behavior: Only retries on test failure, not on skipped tests
• Implementation: RetryTransformer registers RetryAllFailuresAnalyzer
• Logging: Retry attempts are logged in console output

Warning

Retry mechanism only works when tests are executed via testng.xml. Tests run directly without the suite configuration will not have retry functionality.

Usage Example:

@RetrySafe
public class SwaggerPagesTests {
    // All tests in this class will be retried on failure
}

// Or at method level:
@RetrySafe
@Test
public void shouldRetryOnlyThisTest() {
    // Only this test will be retried on failure
}

Test Severity Levels

Tests are categorized using Allure's @Severity annotation to prioritize execution and reporting:

1. BLOCKER: Core functionality that blocks system operation (e.g., authentication, profile creation)
2. CRITICAL: Essential functionality with severe impact (e.g., fetching profiles, app version details)
3. NORMAL: Standard functionality (e.g., updating metadata, configurations)
4. MINOR: Edge cases and error handling scenarios
5. TRIVIAL: UI elements, logging, supplementary features

Usage:

@Test
@Severity(SeverityLevel.CRITICAL)
public void yourTestMethod() {
    // Test implementation
}

Environment-Based Test Exclusion

Tests can be completely excluded from specific environments using @DisabledOnEnv:

@DisabledOnEnv({"prod", "self-hosted"})
@Test
public void testOnlyForDev() {
    // This test will not run in 'prod' or 'self-hosted'
}

Tip

Use @DisabledOnEnv when a test should not appear in reports for certain environments. Use skipTest() when you want the test to appear as skipped in reports.

Test Skipping

Tests can be conditionally skipped based on environment:

@Test
public void testExample() {
    skipTest("Reason for skipping", new String[]{"prod"});
    // Rest of test code
}

cURL and Trace ID Attachment

CurlCaptureFilter records the latest request as a cURL command and stores a lightweight response snapshot per thread. logTraceIdAndGenerateCurl() then adds evidence to the Allure report:

• Attaches cURL for every call where the helper is invoked
• Adds an Allure step with trace ID and observed status
• Attaches HTML response bodies (when content-type is text/html)
• On unexpected status, attaches response status, content type, trace ID, and response body
• On test/configuration failures, TestListener attaches the last cURL, last response snapshot, and exception details

Usage:

Response response = given()
        .headers(headers)
        .body(payload)
        .post(getUrl() + "some/endpoint");

logTraceIdAndGenerateCurl("Create Something", response, HttpStatusCodes.SUCCESS.getValue());

Test Organization

Test Modules

Tests are organized into functional modules:

Build Module

• Commit management
• Profile management
• Queue operations
• Health checks
• Variable groups
• Permissions
• Connections
• Agent management
• Workflows
• History reports
• CodePush
• Activity logs
• Triggers
• Webhooks

Distribution Module

• Profile management
• Testing groups
• Reports
• Group controllers
• Tester API (Profile, Invitation, Device)
• Settings
• Tester Web (Devices, Auth, Files, Home)
• Resign operations

Publish Module

• iOS: Profile, Flow, Actions, App Info, Metadata, Intune Metadata, Resign, Controller
• Android: Profile, Flow, Actions, Play Store App Info, Component, Metadata, Play Store Metadata, Intune Metadata, Resign, Controller
• Variable groups
• Permissions
• Reports (Activity Log, Resign, Publish)

Signing Identity Module

• Permissions
• Certificates
• Keystores
• App Store Connect API
• Provisioning profiles
• Bundle identifiers
• Reports
• CSR (Certificate Signing Request)
• Device management

Organization Module

• Organizations
• API keys
• Credentials (Huawei, App Store Connect, Google Play, Microsoft Intune)
• Authentications (Enterprise Portal, Testing Portal)
• Team activity logs
• Retention reports and retention management
• Domain verification
• Team management
• Notifications (client and webhook notifications)

Enterprise App Store Module

• Profile management
• Permissions
• Portal customization, profile controller, store settings, store API, and general portal API
• Resign operations for iOS and Android enterprise store flows
• Admin operations
• Reports

Other Modules

• Admin (Auth Activity Log, Configurations)
• Dashboard (Summary)
• License
• Reserved Environment
• Swagger Pages (Accessibility)

Test Execution

Tests are executed via TestNG suite configuration (testng.xml):

• Parallel Execution: TestNG runs up to 3 threads with parallel="tests" and thread-count="3"
• Test Groups: Tests are grouped by functional area
• Suite Content: testng.xml is the source of truth for the default suite and currently references 108 test classes
• Listeners:
• TestListener: Console lifecycle logging, failure attachments, setup-failure context, and per-test cURL state cleanup
• RetryTransformer: Handles @DisabledOnEnv exclusions and applies retry analyzer for @RetrySafe tests
• Timeouts: Async build, task, and publish flows use environment-specific seconds from config.properties

Note

The parallel execution with thread-count of 3 helps reduce overall test execution time while maintaining stability. Adjust this value carefully if you encounter resource contention issues.

CI/CD Integration

Jenkins Pipeline (Jenkinsfile)

The Jenkinsfile in this repository automates Docker image build and publish for the test framework. It does not execute the full TestNG suite against every environment in this same file.

1. Build Stage: Builds the Docker image containing the test framework
2. Image Tagging: Tags images based on branch (master and release* branches use latest; develop uses alpha-latest)
3. Registry Push: Pushes images to Google Cloud Artifact Registry
4. Notification: Calls notifySlack(currentBuild.currentResult, currentBuild.durationString) in post { always { ... } }

End-to-end runs that execute mvn clean test ... use the Docker image and the shell scripts under scripts/. Their schedule is controlled by Jenkins jobs, cron definitions, or manual triggers outside this repository.

GitHub Actions

Pull requests trigger Checkstyle through .github/workflows/checkstyle.yaml. The workflow runs mvn checkstyle:checkstyle, converts the XML report to HTML on failure, uploads the report artifact, and comments on the PR with the artifact link.

Docker Integration

Tests run in Docker containers for consistency:

• Base Image: maven:3.9.3-eclipse-temurin-11-focal
• Allure Setup: Includes Allure 2.25.0 for report generation
• Init Process: Uses tini for proper signal handling

Execution Scripts

Shell scripts are provided for running tests in different environments:

• scripts/run-dev-docker-tests.sh
• scripts/run-prep-docker-tests.sh
• scripts/run-prod-docker-tests.sh
• scripts/run-self-hosted-docker-tests.sh
• scripts/run-self-hosted-k8s-docker-tests.sh

Each script runs the apitest image on linux/amd64, mounts the current repository into /app, executes mvn clean test -P{env} -Dsurefire.suiteXmlFiles=testng.xml, and then generates an Allure report with allure generate 'allure-results' --clean.

Schedule, Duration, and Notifications

The exact schedule and Slack targets are operational Jenkins/shared-library configuration, not values stored in this repository. Current operational usage is:

Environment	Frequency / trigger	Typical duration	Slack report channel
dev	No fixed cadence documented in this repository	Varies by run and environment state	Dev and prep API test reports
prep	Run manually before release after the environment is ready	Varies by run and environment state	Dev and prep API test reports
prod	Runs every day at 05:15	Usually 1h-1h30m.	Prod API test reports
self-hosted	Run manually before SH release after the environment is ready	Varies by run and environment state	SH API test reports

The checked-in Jenkinsfile still only defines Docker image build/push behavior. Scheduled or manual API test execution is handled by Jenkins jobs or other operational automation outside this repo.

Known timeout values from checked-in environment configs:

Environment	buildStatusTimeout	taskStatusTimeout	publishStatusTimeout
dev	2700s / 45m	1200s / 20m	2700s / 45m
prep	2700s / 45m	2700s / 45m	2700s / 45m
prod	1800s / 30m	1200s / 20m	1800s / 30m
self-hosted	2700s / 45m	1200s / 20m	2700s / 45m
self-hosted-k8s	2700s / 45m	1200s / 20m	2700s / 45m

Code Quality

Checkstyle

Code quality is enforced through Checkstyle using Google's Java style guide:

• Configuration: google_checks.xml
• Local / Maven lifecycle: maven-checkstyle-plugin is bound to the validate phase (mvn validate or a later phase runs checkstyle:check)
• Pull requests: GitHub Actions runs mvn checkstyle:checkstyle and uploads an HTML report on failure
• Auto-fix: OpenRewrite plugin can automatically fix many issues

Commands:

# Check for issues
mvn checkstyle:check

# Auto-fix issues
mvn rewrite:run

Pull Request Checklist

All PRs must ensure:

• README.md updated
• Code comments added
• Tested in all relevant environments
• Code formatting applied (mvn rewrite:run)
• No magic numbers
• Code duplication checked
• Appropriate severity levels assigned

Required Before Merge

All checklist items must be completed before a PR can be merged. The PR template includes these items for verification.

Reporting

Allure Reports

Test results are published to an Allure report server:

• Server: Hosted on AppCircle Git macOS server (77.92.124.11)
• Base URL: https://api-test-reports.appcircle.io
• Structure: Reports organized by environment and build number
• Access Pattern: https://api-test-reports.appcircle.io/{env}/{build-number}
• Username: appcircle
• Password: acreport1

Report Structure

reports/
├── dev/
│   ├── 1/
│   ├── 206/
│   └── ...
└── prod/
    ├── 1/
    └── ...

Best Practices

Writing Tests

1. Extend AbstractBaseTest: All test classes should extend com.appcircle.apitest.modules.AbstractBaseTest
2. Use BDD Format: Test method names should follow BDD conventions (e.g., shouldCreateProfileWhenValidDataProvided)
3. Assign Severity: Every test should have an appropriate severity level
4. Handle Cleanup: Ensure test data is cleaned up after execution
5. Use Helper Methods: Leverage base class methods for common operations
6. Log Trace IDs: Always call logTraceIdAndGenerateCurl() after requests

Test Data Management

• Use uuidGenerator() for unique identifiers
• Use generateRandomStringWithPrefix() for test data with timestamps
• Clean up created resources in @AfterMethod or @AfterClass

Environment Configuration

• Store environment-specific values in config.properties
• Use system properties for runtime configuration
• Avoid hardcoding environment-specific values

Error Handling

• Always validate response status codes
• Use appropriate HTTP status code constants
• Attach relevant debugging information on failure

Local Development

Prerequisites

• Java 11+
• Maven 3.9+
• Docker (for containerized execution)

Running Tests Locally

Local Preview

Always preview your changes locally before pushing. This helps catch issues early and ensures documentation renders correctly.

x86_64 Architecture:

# Build image
docker image build -t apitest .

# Run tests for dev environment
docker run --name appcircle-api-test apitest /bin/bash -c \
  "mvn clean test -Pdev -Dsurefire.suiteXmlFiles=testng.xml; \
   allure generate 'allure-results' --clean"

# Copy results
docker cp appcircle-api-test:/app/allure-report .

# Cleanup
docker rm appcircle-api-test

Container Cleanup

Remember to remove the container after copying results. The container name appcircle-api-test will conflict if you try to run tests again without removing it first.

ARM64 (M1/M2 Mac):

# Create builder
docker buildx create --use

# Build image
docker buildx build --platform linux/amd64 -t apitest --output type=docker .

# Run tests
docker run --platform linux/amd64 --name appcircle-api-test apitest \
  /bin/bash -c "mvn clean test -Pdev -Dsurefire.suiteXmlFiles=testng.xml; \
   allure generate 'allure-results' --clean"

Using Remote Image

docker run --name appcircle-api-test \
  europe-west1-docker.pkg.dev/appcircle/docker-registry/api-automation-test:latest \
  /bin/bash -c "mvn clean test -Pdev -Dsurefire.suiteXmlFiles=testng.xml; \
   allure generate 'allure-results' --clean"

Configuration

Environment Properties

Each environment's config.properties contains environment-specific URLs, credentials, IDs, and timeout values. Important shared keys include:

• url: Base API URL
• authUrl: Authentication service URL
• pat: Personal Access Token for authentication
• buildStatusTimeout: Seconds to wait while polling build status
• taskStatusTimeout: Seconds to wait while polling task status
• publishStatusTimeout: Seconds to wait while polling publish status

Maven Profiles

Profiles are defined in pom.xml:

• dev (default)
• prep
• prod
• self-hosted
• self-hosted-k8s

Troubleshooting

Common Issues

Container name conflict:

docker rm appcircle-api-test

Test failures: - Check Allure reports for detailed error information - Verify trace IDs in reports for API debugging - Review cURL commands attached to failed tests

Environment-specific failures: - Verify config.properties for the target environment - Check PAT validity and permissions - Ensure environment URLs are correct

Related Documentation

For more details on the API automation test project, refer to the api-automation-test repository:

• api-automation-test - Source code and documentation
• Test Scenarios - Detailed BDD scenarios for each module
• README - Project setup and usage instructions