Marketplace Plugin Testing¶

Overview¶

Appcircle publishes plugins for four external CI/CD ecosystems so that teams can distribute their applications without using an Appcircle build. Each ecosystem has two plugins, one for each product line:

Testing Distribution (TD) uploads an APK, AAB, or IPA to a Testing Distribution profile.
Enterprise App Store (EAS) uploads a binary to the Enterprise App Store.

This makes eight plugins in total. They are published to four separate marketplaces, released independently of the Appcircle platform, and executed on infrastructure that Appcircle does not control. A regression in any of them is visible to customers immediately and cannot be rolled back by a platform deployment, so every plugin is validated end to end against a real Appcircle Cloud organization before a release is relied upon.

This document describes how that validation is performed: the shared methodology, how to select the plugin version under test in each ecosystem, the per-ecosystem runbooks, the verification checklist, and the pitfalls that have already cost time in previous rounds.

The methodology was established in PL-310 and its per-ecosystem sub-issues (PL-311 GitHub Actions, PL-312 Azure DevOps, PL-313 Fastlane, PL-314 Jenkins), and extended while verifying the fixes tracked in BE-9028.

Scope¶

Plugins covered¶

Ecosystem	Product line	Listing	Source repository
Fastlane (RubyGems)	TD	`fastlane-plugin-appcircle_testing_distribution`	`appcircleio/fastlane_plugin_appcircle_testing_distribution`
Fastlane (RubyGems)	EAS	`fastlane-plugin-appcircle_enterprise_app_store`	`appcircleio/fastlane-plugin-appcircle_enterprise_app_store`
GitHub Actions	TD	Appcircle Testing Distribution	`appcircleio/appcircle-testing-distribution-githubaction`
GitHub Actions	EAS	Appcircle Enterprise App Store	`appcircleio/appcircle-enterprise-app-store-githubaction`
Jenkins	TD	`appcircle-testing-distribution`	`jenkinsci/appcircle-testing-distribution-plugin`
Jenkins	EAS	`appcircle-enterprise-store`	`jenkinsci/appcircle-enterprise-app-store-plugin`
Azure DevOps	TD	`Appcircle.build-release-task`	`appcircleio/appcircle-azuredevops-testdist-task-extension`
Azure DevOps	EAS	`Appcircle.enterprise-app-store`	`appcircleio/appcircle-azuredevops-entstore-task-extension`

Not covered yet¶

The newer Publish plugins (fastlane-plugin-appcircle_publish, appcircle-publish-githubaction, jenkinsci/appcircle-publish-plugin, appcircle-azuredevops-publish-task-extension) are not part of the test rounds described here. When they are added, they follow the same methodology: pick the version under test, run the ecosystem runbook, verify against a real profile, record the result.

Test methodology¶

Five rules define how these tests are run. Follow them so results stay comparable between rounds and between ecosystems.

1. Test the plugin, not the build. No application is compiled during the test. Each pipeline pulls a pre-built, known-good binary with curl and hands its path to the plugin. This isolates the plugin from build, signing, and toolchain failures, and keeps a full round short.

APK: https://storage.googleapis.com/appcircle-dev-common/self-hosted/sample-apps/appcircle-sample-flutter.apk
IPA: https://storage.googleapis.com/appcircle-dev-common/self-hosted/sample-apps/appcircle-sample-ios.ipa

2. Test against a real environment. Uploads go to real profiles in Appcircle production, in the appcircleteam SSO organization (9a647794-62c4-4be5-82d7-5f4505a75c65). Mock endpoints would not exercise authentication, the multipart upload contract, or the commit step, which is where these plugins have historically failed.

3. Cover both product lines and both platforms. Every round runs four uploads per ecosystem: TD with an APK, TD with an IPA, EAS with an APK, EAS with an IPA.

4. Know exactly which build you tested. Each ecosystem installs plugins differently, and in every one of them the default path installs the released version, not your branch. Selecting the version under test is a deliberate step, documented per ecosystem below. Always record the version or commit SHA next to the result.

5. Record the result where the next person will look. Findings go into a comment on the matching Linear issue, not into a private note. A result is only useful if it states the date, the plugin version or commit, the parameters used, and what was not covered.

Common prerequisites¶

Authentication¶

All eight plugins authenticate with an Appcircle Personal API Token (PAT).

An API key is not accepted

Passing a plain API key makes the plugin fail at login. Generate a Personal API Token in the Appcircle dashboard and use that. If a run fails during authentication, regenerate the token and update the secret before investigating anything else.

The token is always supplied through the ecosystem's secret mechanism, never written into a pipeline file:

Ecosystem	Secret mechanism	Name used in the test setups
Fastlane	Environment variable, or an Appcircle environment variable group	`AC_PERSONAL_API_TOKEN`
GitHub Actions	Repository secret	`AC_MARKETPLACE_PERSONNEL_ACCESS_KEY`
Jenkins	Credentials (secret text)	`MARKETPLACE_TEST_PAT`
Azure DevOps	Secret pipeline variable, variable group `Marketplace-Test`	`AC_PERSONAL_API_TOKEN`

Target profiles¶

The following profiles in the appcircleteam organization are the standing test targets. Reuse them instead of creating new ones, so a round can be compared with the previous round.

Testing Distribution profile: https://my.appcircle.io/distribute/detail/78de3b05-b08e-4564-b76f-fccdebb9d4d9?org=9a647794-62c4-4be5-82d7-5f4505a75c65
Enterprise App Store, iOS: https://my.appcircle.io/enterprise-store/profiles/317f1758-186a-4a20-8ab8-2412b18ca6da?org=9a647794-62c4-4be5-82d7-5f4505a75c65
Enterprise App Store, Android: https://my.appcircle.io/enterprise-store/profiles/b3cdaf7b-6a1c-48e9-9f46-7671e641dba2?org=9a647794-62c4-4be5-82d7-5f4505a75c65

Some ecosystems historically used their own TD profile name (Marketplace-Github-Action-Test, Marketplace-Fastlane-Test, Jenkins Plugin Test). Either is acceptable as long as the profile name in the pipeline matches an existing profile, or createProfileIfNotExists is set to true.

Cloud only¶

The Fastlane, GitHub Actions, and Jenkins plugins support Appcircle Cloud only and expose no endpoint inputs. Only the Azure DevOps extensions accept authEndpoint and apiEndpoint, which default to https://auth.appcircle.io and https://api.appcircle.io. Do not plan self-hosted coverage for the other three ecosystems until endpoint inputs exist there, and re-confirm this limitation each round, because it is a documented product statement that changes over time.

Selecting the version under test¶

This is the step that is most often done wrong, and getting it wrong invalidates the whole round: the pipeline passes, but it exercised the previously released plugin rather than the change you wanted to verify.

Fastlane¶

fastlane add_plugin <name> installs the latest stable release from RubyGems. Prerelease versions are not installed by default, so a fix that was only published as a beta, or that only exists on develop, is not what runs.

To test a branch, pin the plugins to git in fastlane/Pluginfile instead of using add_plugin:

# fastlane/Pluginfile
gem 'fastlane-plugin-appcircle_testing_distribution', git: 'https://github.com/appcircleio/fastlane_plugin_appcircle_testing_distribution', branch: 'develop'
gem 'fastlane-plugin-appcircle_enterprise_app_store', git: 'https://github.com/appcircleio/fastlane-plugin-appcircle_enterprise_app_store', branch: 'develop'

Then resolve and run through Bundler so the git sources are actually used:

bundle install
bundle exec fastlane td_test
bundle exec fastlane eas_test

Points to watch:

Delete Gemfile.lock (and fastlane/Pluginfile.lock if present) before bundle install when a previous run resolved the released gems, otherwise the old versions stick.
Bundler prints the git source and revision during install. Read it back and record the revision with the result.
To test a specific published prerelease instead of a branch, pin the exact version: gem 'fastlane-plugin-appcircle_testing_distribution', '0.5.1.beta.1'.
If the run is executed through the Appcircle build profile described below, the Custom Script step generates the Pluginfile itself, so the same edit has to be made there.

GitHub Actions¶

There is no install step. An action always runs the ref pinned in the workflow file, so the version under test is changed by editing the uses: line in the test repository's workflow and pushing that change. Without a push there is no new run.

- uses: appcircleio/appcircle-testing-distribution-githubaction@develop   # branch
- uses: appcircleio/appcircle-enterprise-app-store-githubaction@<commit-sha>   # reproducible

Points to watch:

The run's "Set up job" section lists the resolved action versions. Pinning a commit SHA makes the result unambiguous.
These are JavaScript actions, so the branch you point at must contain the built output the entrypoint loads (dist/). Source changes that were never built and committed run as stale code or fail to resolve the entrypoint.
Restore the refs to the released version after the test, so the test workflow stops tracking a development branch.
Validate the released tag as well, not only develop. Customers consume the Marketplace listing.

Jenkins¶

Jenkins plugins are Maven projects packaged as .hpi. Build the branch, then deploy the artifact manually.

git clone -b develop https://github.com/jenkinsci/appcircle-testing-distribution-plugin.git
cd appcircle-testing-distribution-plugin
git rev-parse --short HEAD          # record this SHA with the result
mvn -DskipTests clean package       # produces target/*.hpi

Install it through Manage Jenkins > Plugins > Advanced settings > Deploy Plugin, then restart Jenkins (/safeRestart). Repeat for jenkinsci/appcircle-enterprise-app-store-plugin.

Points to watch:

Requires JDK 17 or 21 and Maven 3.9 or newer, because the plugin baseline is Jenkins 2.440.3. There is no Maven wrapper in the repository.
Manual deployment does not resolve dependencies. Install json-api and apache-httpcomponents-client-4-api from the Update Center first if they are missing.
The pom.xml sets changelist to 999999-SNAPSHOT, so a locally built plugin appears with that version. This is useful: it is higher than any released version, so the Update Center cannot silently replace it, and it is an unmistakable signal in the Installed list that the controller is running a test build.
Confirm under Manage Jenkins > Plugins > Installed that the version is the snapshot, not the Marketplace release, before trusting a result.
Deploying a snapshot affects every job on that controller. Restore the released version when the round is finished.

Azure DevOps¶

An Azure DevOps task comes from an installed Marketplace extension, so a branch cannot be referenced from the pipeline. Testing a branch means building a private "Dev" extension from that branch and installing it into the organization. Both repositories already carry the override for this in configs/dev.json (a separate extension id and "public": false).

git clone -b develop https://github.com/appcircleio/appcircle-azuredevops-testdist-task-extension.git
cd appcircle-azuredevops-testdist-task-extension
yarn install && (cd buildandreleasetask && yarn install)   # EAS repository uses task/
yarn package
npx tfx extension create --manifest-globs vss-extension.json --overrides-file configs/dev.json
npx tfx extension publish --manifest-globs vss-extension.json --overrides-file configs/dev.json \
  --token "$MARKETPLACE_PAT" --share-with appcircleio

Points to watch:

Task GUID collision. The dev package still carries the production task id (TD ffeb39c1-fd80-4333-b572-9de0f1e3dae1, EAS b8941f3f-aee4-4e0f-849b-eb19c26ce729) and the production task name. Two extensions with the same task id cannot coexist in one organization. For a dev build, change the id in task.json to a fresh GUID and give name and friendlyName distinct values, or uninstall the production extension first.
Version numbers. The Marketplace rejects an equal or lower version. pipeline.sh derives the version from the latest git tag and publishes only from main, so a manual build needs the versions in vss-extension.json and task.json bumped by hand.
The publish token is a Visual Studio Marketplace PAT with Manage or Publish permission on the Appcircle publisher.
Update the task reference in the pipeline YAML to the dev task name, for example AppcircleTestingDistributionDev@0.

To test a released version rather than a branch, no packaging is needed, but the version still does not arrive on its own: install or update the extension in the organization from the Visual Studio Marketplace first, then point the pipeline at it.

Testing Distribution: https://marketplace.visualstudio.com/items?itemName=Appcircle.build-release-task
Enterprise App Store: https://marketplace.visualstudio.com/items?itemName=Appcircle.enterprise-app-store

The pipeline pins the extension versions explicitly, so update them in azure-pipelines.yml (see the runbook below) after installing. A pipeline referencing a version that is not installed in the organization fails to resolve the task.

Per-ecosystem runbooks¶

Fastlane¶

Two ways to run the test. Option A is the standing setup; Option B is for debugging a plugin locally.

Option A, from an Appcircle build profile. A build profile runs the whole lane with no local Ruby setup:

Environment: Appcircle production, Development Team sub-organization.
Build profile Fastlane Marketplace Test, workflow Fastlane Marketplace Test, Default Configuration.
Quick link: https://my.appcircle.io/build/detail/34039da2-3083-4cfb-b022-8cf62a76bab0?org=9a647794-62c4-4be5-82d7-5f4505a75c65

The workflow's Custom Script step installs Fastlane and both plugins through a Gemfile and Pluginfile, curls the sample APK and IPA, and runs the marketplace_test lane (TD and EAS for both artifacts). The PAT comes from the profile's environment variable group. The step turns red if any upload fails.

Option B, locally.

brew install fastlane            # or: gem install fastlane
mkdir ac-fastlane-test && cd ac-fastlane-test && mkdir fastlane
fastlane add_plugin appcircle_testing_distribution
fastlane add_plugin appcircle_enterprise_app_store

Define td_test and eas_test lanes that curl the artifacts and call appcircle_testing_distribution and appcircle_enterprise_app_store with personalAPIToken, appPath, and the product-specific inputs. Export AC_PERSONAL_API_TOKEN, AC_PROFILE_NAME, APK_URL, and IPA_URL in the same shell, then run both lanes. Dir.pwd inside a lane is the fastlane/ directory, so download the artifacts there and pass full paths.

The reference lanes and full parameter list are in PL-313.

GitHub Actions¶

Test repositories: appcircleio/appcircle-sample-ios (IPA) and appcircleio/appcircle-sample-android (APK).
Branch: feature/pl-311-marketplacetest.
Workflow: .github/workflows/ac-plugin-test.yml, triggered on push to that branch, with one TD job and one EAS job. Each job curls the artifact and runs the action with personalAPIToken: ${{ secrets.AC_MARKETPLACE_PERSONNEL_ACCESS_KEY }}.

To re-run: set the action refs to the version under test, commit, and push to the branch. The push starts the run. If the file is otherwise unchanged, alter a trivial line so there is something to commit. Watch the run under the repository's Actions tab.

Azure DevOps¶

Organization: https://dev.azure.com/appcircleio, project Appcircle Marketplace Test.
Pipeline definition: https://dev.azure.com/appcircleio/_git/Appcircle%20Marketplace%20Test?path=/azure-pipelines.yml
Cloud Azure DevOps is used, not a self-hosted instance.
Agent: there is no hosted runner available to this project. The Microsoft-hosted default pool requires a parallelism grant request, which has not been filed, so a self-hosted agent connected to the Default pool with a PAT is used instead. The last round ran on a self-hosted macOS machine.

Confirm an agent is online before starting a round

The self-hosted agent is a personal machine, not permanent infrastructure, so it is frequently offline. A queued pipeline simply waits and the round stalls. Check the pool first: https://dev.azure.com/appcircleio/Appcircle%20Marketplace%20Test/_settings/agentqueues?queueId=11&view=agents

If no agent is online, either ask berk@appcircle.io to connect one, or connect your own machine: choose New agent at the top right of that page and follow the instructions. It takes about ten minutes.

The PAT comes from the Marketplace-Test variable group as a secret variable, and artifacts are pulled into $(Build.ArtifactStagingDirectory).

The pipeline runs AppcircleTestingDistribution@0 for the APK and the IPA, then AppcircleEnterpriseStore@0 for both. A reference copy of the YAML is in PL-312; the version that actually runs lives in the repository linked above.

The extension version is pinned in the pipeline file

azure-pipelines.yml declares which extension version each task uses, so editing that file is how the version under test is selected. A new version has to be installed into the organization from the Visual Studio Marketplace before the pipeline can reference it, otherwise the task does not resolve. Install or update the extension first, then update the version in azure-pipelines.yml and commit.

Jenkins¶

A parameterized job is already set up, so a re-test is one build:

https://jenkins.appcircle.io/job/marketplace-plugins/job/appcircle-jenkins-plugin-test/job/Appcircle%20Testing%20Distribution%20Jenkins%20Plugin%20Test/

Agent label agent (Linux), which satisfies the Linux agent coverage requirement.
Credential MARKETPLACE_TEST_PAT (secret text) is consumed through environment { ... credentials(...) }, so Jenkins masks it in the log.
Parameters: AC_PROFILE (TD profile name), PUBLISH_TYPE (0 None, 1 Beta, 2 Live), APK_URL, IPA_URL.

pipeline {
  agent { label 'agent' }
  parameters {
    string(name: 'AC_PROFILE',  defaultValue: 'Jenkins Plugin Test')
    choice(name: 'PUBLISH_TYPE', choices: ['0','1','2'], description: '0 None / 1 Beta / 2 Live')
    string(name: 'APK_URL', defaultValue: 'https://storage.googleapis.com/appcircle-dev-common/self-hosted/sample-apps/appcircle-sample-flutter.apk')
    string(name: 'IPA_URL', defaultValue: 'https://storage.googleapis.com/appcircle-dev-common/self-hosted/sample-apps/appcircle-sample-ios.ipa')
  }
  environment { MARKETPLACE_TEST_PAT = credentials('MARKETPLACE_TEST_PAT') }
  stages {
    stage('Download artifacts') {
      steps { sh 'curl -fSL -o app.apk "$APK_URL"'; sh 'curl -fSL -o app.ipa "$IPA_URL"' }
    }
    stage('TD') {
      steps {
        appcircleTestingDistribution personalAPIToken: MARKETPLACE_TEST_PAT, profileName: "${params.AC_PROFILE}",
          createProfileIfNotExists: true, appPath: 'app.apk', message: 'TD - APK'
        appcircleTestingDistribution personalAPIToken: MARKETPLACE_TEST_PAT, profileName: "${params.AC_PROFILE}",
          createProfileIfNotExists: true, appPath: 'app.ipa', message: 'TD - IPA'
      }
    }
    stage('EAS') {
      steps {
        appcircleEnterpriseAppStore personalAPIToken: MARKETPLACE_TEST_PAT, appPath: 'app.apk',
          summary: 'EAS - APK', releaseNotes: 'plugin re-test', publishType: params.PUBLISH_TYPE
        appcircleEnterpriseAppStore personalAPIToken: MARKETPLACE_TEST_PAT, appPath: 'app.ipa',
          summary: 'EAS - IPA', releaseNotes: 'plugin re-test', publishType: params.PUBLISH_TYPE
      }
    }
  }
}

Also exercise the plugin as a freestyle build step at least once per round, so both usage modes are covered.

Verification checklist¶

A green pipeline is necessary but not sufficient. Check all four levels before calling a round passed.

Log level

The plugin step reports a successful login, then a successful upload for each artifact. The TD plugins print the upload information call, the upload, the commit, and the resulting task id; the EAS plugins print less (see the known issues below).
The token does not appear anywhere in plain text. Each ecosystem masks differently: GitHub Actions shows personalAPIToken: ***, Jenkins prints "Masking supported pattern matches of $MARKETPLACE_TEST_PAT". Confirm the masking rather than assuming it.
No new deprecation or runtime warning compared to the previous round.

Platform level

The APK and the IPA are visible as new versions in the target TD profile.
The APK and the IPA are visible in the matching EAS profile, one per platform.
If createProfileIfNotExists was used, the profile was actually created and the upload landed in it.

Device level

Download at least one artifact from the profile and install it on a device. An upload can succeed while producing an unusable artifact.

Documentation level

The inputs behave as documented, including createProfileIfNotExists and publishType.
The listing name makes it obvious whether the plugin is TD or EAS.
Note anything the plugin documentation does not cover, or covers incorrectly.

Known pitfalls¶

These have already cost time. Read them before starting a round.

The Enterprise App Store rejects the same binary twice¶

EAS does not overwrite an existing version with an identical binary, and the test always uploads the same sample APK and IPA. Delete the previously uploaded version from both EAS profiles before every re-run, otherwise the EAS stage fails or appears to succeed while the store still holds the old version. Testing Distribution does not have this restriction and accepts the same binary as a new version.

`publishType` Beta and Live fail with the sample apps¶

With publishType 1 (Beta) or 2 (Live), EAS actually publishes and validates the binary, and the sample apps are expired, so the run fails with EnterpriseStoreServer.AppVersion.NotValid ("The app version has expired and cannot be published"). Use 0 (None) for routine rounds and note that Beta and Live are uncovered. Exercising them properly requires a valid, non-expired signed build.

A Personal API Token is required¶

An API key is rejected. See the authentication note above.

Verify which version actually ran¶

Every ecosystem installs the released version by default. A round that does not explicitly pin a branch, tag, or SHA has tested the release, not the change. Record the version or commit next to the result. This gap was visible in the Jenkins round: the build log did not print plugin versions, so the tested build was not traceable from the log alone.

EAS logging is thinner than TD¶

The EAS plugins print only a login line and a final success line, while the TD plugins print each step of the upload with its status code. This is a real difference in error surfacing, not only cosmetics: a failing intermediate step in EAS gives the tester almost nothing to work with. The Fastlane EAS plugin was fixed under BE-9003 by adding per-step UI.message / UI.success output and 200 to 299 status validation. The Jenkins EAS plugin still shows the same gap.

Runtime deprecations age out silently¶

Two rounds surfaced runtime deprecations that would eventually break the plugins on the providers' runners:

Azure DevOps tasks warned that Node 16 is end of life. The develop branches now declare Node20_1; confirm the warning is gone.
The GitHub Actions plugins warned that Node 20 is deprecated and were being forced onto Node 24, and emitted DEP0169 for url.parse(). These are tracked as BE-9000 and BE-9001.

Treat every new warning line in a run log as a finding, even when the run is green.

Loading two plugins together can collide¶

The two Fastlane plugins each defined a top level BASE_URL constant, so loading them in the same run produced warning: already initialized constant BASE_URL. Fixed under BE-9002 by scoping the constant inside each plugin's module. Because the test runs TD and EAS in one process, this class of collision only shows up here, which is exactly why both plugins are exercised together.

Azure DevOps has no standing runner¶

Using the Microsoft-hosted pool requires a parallelism grant form that has not been submitted, so every Azure DevOps round depends on a self-hosted agent that someone connects from their own machine. Check the pool before scheduling a round; if nothing is online the pipeline queues indefinitely. Connecting a new agent takes about ten minutes, and the details are in the Azure DevOps runbook above.

Deploying a test build affects everything on that Jenkins controller¶

The Jenkins job runs on the shared controller. A 999999-SNAPSHOT plugin is installed for the whole instance, not for one job. Coordinate before deploying, and restore the released version when the round is over.

Recording results¶

Work is tracked in Linear under the Marketplace Plugins project:

PL-310 is the umbrella issue for a full round.
PL-311, PL-312, PL-313, and PL-314 hold the per-ecosystem runbooks and results.
Fixes found during a round become their own issues (BE-9000 to BE-9003), and verification of a group of fixes is tracked in a single test issue such as BE-9028 rather than one sub-issue per fix.

A result comment states:

Run date and environment, in Türkiye time.
The plugin version or commit SHA actually exercised, and how it was pinned.
The parameters used, including publishType and the target profile.
Pass or fail per step, with the evidence quoted from the log.
Findings, separated into defects and observations.
What was not covered, so the next round knows where to start.

Keeping the "not covered" list honest is what makes these rounds cumulative. As of the last round, the open items are: publishType Beta and Live coverage, the Jenkins freestyle build step usage mode, device installation of a downloaded artifact, validating released tags rather than development branches, and the Publish plugin family.

Marketplace Plugin Testing¶

Overview¶

Scope¶

Plugins covered¶

Not covered yet¶

Test methodology¶

Common prerequisites¶

Authentication¶

Target profiles¶

Cloud only¶

Selecting the version under test¶

Fastlane¶

GitHub Actions¶

Jenkins¶

Azure DevOps¶

Per-ecosystem runbooks¶

Fastlane¶

GitHub Actions¶

Azure DevOps¶

Jenkins¶

Verification checklist¶

Known pitfalls¶

The Enterprise App Store rejects the same binary twice¶

publishType Beta and Live fail with the sample apps¶

A Personal API Token is required¶

Verify which version actually ran¶

EAS logging is thinner than TD¶

Runtime deprecations age out silently¶

Loading two plugins together can collide¶

Azure DevOps has no standing runner¶

Deploying a test build affects everything on that Jenkins controller¶

Recording results¶

`publishType` Beta and Live fail with the sample apps¶