Acceptance Testing

Manifold makes it easy for providers to validate their integration with a command line tool we call Grafton. Grafton is used for local development of a Provider's implementation of our Provider API endpoints.

Acceptance testing data flow

Grafton works by standing in for the Manifold API and sends various requests to your in-progress integration. These tests include: resource and credential provisioning, resizing, deprovisioning, and single sign-on requests. To catch implementation errors, Grafton not only tests the success path but also an integration's validation and error handling.

While the tests are running through Grafton, a local version of Manifold's Connector API is also run for retrieving user or resource data as well as completing asynchronous operations.

Grafton is open source with releases available on GitHub.

We appreciate and welcome all feedback as you use our development tooling.

Installation

The latest installation instructions are documented in Grafton's GitHub repository.

Configuration

Authenticating Manifold requests

The first step to use Grafton is to generate a test master key which is used for signing requests handled by your integration.

When testing your integration, Grafton authenticates using this test master key. When running in production, Manifold will authenticate using our production public key.

$ grafton generate
Generating Master Keypair
Writing master keypair to file: masterkey.json
Success.

$ cat masterkey.json
{
  # Use the public_key in your integration
  "public_key": "bBbFBCDG8UEqSgiygslX2TZP/RUCD0GTHh43nXFxx1U=",
  # Grafton will use the private_key to sign requests
  "private_key": "s/4ZT/a4BXBrffQdC7bbW+NalAVc7THEuctqeruRVj1sFsUEIMbxQSpKCLKCyVfZNk/9FQIPQZMeHjedcXHHVQ=="
}

Use the public master key contained in the generated file to validate requests created by Grafton during testing. Once integrated and in production, you must be using Manifold's Master Public Signing Key to validate requests coming from the Manifold servers.

Connector API (Optional)

To make requests against the locally run Connector API served by Grafton, you can use the following set of test credentials:

CLIENT_ID=21jtaatqj8y5t0kctb2ejr6jev5w8
CLIENT_SECRET=3yTKSiJ6f5V5Bq-kWF0hmdrEUep3m3HKPTcPX7CdBZw

These values will be consumed by your application, and must also be supplied to the Grafton test command, as shown:

$ grafton test \
    --client-id=21jtaatqj8y5t0kctb2ejr6jev5w8 \
    --client-secret=3yTKSiJ6f5V5Bq-kWF0hmdrEUep3m3HKPTcPX7CdBZw \
    ... # other command line flags

Running Tests

Once you've configured your integration with the generated signing key and, optionally, with OAuth credentials, you can begin testing with Grafton.

You can invoke the test command with parameters representing how a user may be configuring your product on Manifold. Typically you need to provide the product, plan, and a region.

In this example we look at resizing a resource. We start with a resource of the plan small from the bear product and are resizing to the large plan:

$ grafton test \
    --product=bear \             # Product label
    --plan=small \               # Plan label
    --new-plan=large \           # Plan label to resize to once the resource has been provsisioned
    --plan-features='{"age":2,"hat_color":"red"}' \  # Features of the plan
    --region=aws::us-east-1 \    # Region label
    --client-id=21jtaatqj8y5t0kctb2ejr6jev5w8 \                   # OAuth Credential (Optional)
    --client-secret=3yTKSiJ6f5V5Bq-kWF0hmdrEUep3m3HKPTcPX7CdBZw \ # OAuth Credential (Optional)
    --connector-port=3000 \      # Port used by Grafton to serve Connector API (Optional)
    http://localhost:4567/v1     # URL to your integration

Green checkmarks will be shown once for each passing test as shown here:

Grafton output example

Reference

Use the --help flag to see all available Grafton options:

$ grafton --help

NAME:
   grafton - Tool for testing integrations with Manifold

USAGE:
   grafton [global options] command [command options] [arguments...]

VERSION:
   0.15.0

COMMANDS:
     credentials  Manage OAuth 2 credential pairs for Manifold.co
     generate     Generates public and private signing keys for testing Manifold API integrations locally
     serve        Serves a local version of of the Connector API
     test         Tests the API endpoints required to integrate with Manifold
     version      Display version of utility
     help, h      Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

Here is an example of viewing the parameters of the test command:

$ grafton test --help

NAME:
   grafton test - Tests the API endpoints required to integrate with Manifold

USAGE:
   grafton test [command options] [url]

OPTIONS:
   --product value            The label of the product being provisioned [$PRODUCT]
   --plan value               The label of the plan for the provisioning resource [$PLAN]
   --plan-features value      A JSON object describing the selected features for the provisioning resource [$PLAN_FEATURES]
   --new-plan value           The plan to resize the instance to from the original plan [$NEW_PLAN]
   --new-plan-features value  A JSON object describing the selected features for the resizing from the original plan [$NEW_PLAN_FEATURES]
   --region value             The label of the region which the resource will be provision in [$REGION]
   --import-code value        The import code to import an existing resource for that resource [$IMPORT_CODE]
   --exclude value            Exclude running these feature tests (and those that depend on it) [$EXCLUDE]
   --no-error-cases           Skip running the error case tests [$NO_ERROR_CASES]
   --log value                Informational logging level during tests. One of (off, info, verbose) (default: "off") [$LOG]
   --client-id value          Client ID to use for SSO and local Connector API testing [$OAUTH2_CLIENT_ID]
   --client-secret value      Client secret to use for SSO and local Connector API testing [$OAUTH2_CLIENT_SECRET]
   --connector-port value     Local port for running the fake Connector API for SSO and Async testing (default: 0) [$CONNECTOR_PORT]
   --callback-timeout value   duration to wait (max. 24hours) for a callback (default: 5m) [$CALLBACK_TIMEOUT]
   --resource-measures value  Optional measures map to be returned by resource measures (default: "{\"feature-a\": 0, \"feature-b\": 1000}")
   --credential value         Describes the credential type that is supported by this product. One of (single, multiple) (default: "multiple")