Developer Console
Sign In
Sign In
Developer Console

Catalog Ingestion API

Prerequisites

In order for a catalog to be set up, the following must already exist:

  • The Store(s) for which the catalog is being set up, must already exist
  • Store login credentials which include a user with catalog upload permissions
  • Complete the Tax Interview and State tax registration process if applicable

Catalog API Solution Overview

The Catalog API provides a programmatic way to insert and update existing catalog items. The data flow design is a representative sample of the business workflows typically used in the retail space.


Prepare your catalog

  • Define the process to update the catalog in the Amazon systems. It is recommended to look into how frequent your catalog changes. The frequency will inform the architecture decision and data flow for the catalog updates. For detailed description, review Considerations

  • Identity the catalog data store. You will need to have access to the catalog updates. This might require to work with the security team to ensure access is granted and the catalog data is secured.

  • The Amazon Catalog API provides you with a set of required fields to upload a catalog. The values might be named differently in your catalog system. You are required to ensure that the correct data elements are mapped between you catalog and the Amazon Catalog API.

Setup your environment

  • AWS Account: This pattern requires an AWS account. For instructions on how to get an AWS account, please see here

  • Completed the Amazon onboarding: Retailer onboarding must be completed before the implementation of this solution

  • AWS Service onboarding: you will be working with the Amazon team to onboard your Catalog API to the Amazon systems. The process will include steps such as providing an IAM role ARN that will be used to call the Amazon API. The Amazon team will provide API invocation endpoints based on the AWS region and the Amazon product you are utilizing for your store. If you elect to use the solution in the Target Architecture section, you will also be required to provide an AWS SQS ARN and follow steps to confirm subscription to the SQS service. For steps on how to confirm SQS subscriptions, please see AWS docs here.

Develop the integration

The following reference architecture is provided as a guide for implementing the catalog API. For detailed step by step instructions visit the Amazon merchant portal


This sample solution follows the following steps:

Catalog upload status in this sample solution is provided through an SNS topic. For an example of Catalog upload status check using API see full solution guide in the Amazon merchant portal

1.Customer will start the process by calling an AWS Lambda function to start the Catalog Upload Note: You are allowed to perform 10 requests per second and max of 10,000 items per upload

2.The Lambda will acquire STS credentials from an AWS IAM role to authorize the operations to manage the catalog upload process. Note: Please follow your organization best practices for least privilege permissions to the IAM role while will providing the necessary authorization to perform the operations required to manage the process to upload the catalog. Please ensure the IAM role has the Invoke API Policy. Sample policy provided in the AWS Documenation. Failure to attach the correct permissions to the IAM role will result in a 403 Error when calling the Amazon API

3.A Lambda will trigger and manage the catalog upload process. The Step Function will perform three key operations illustrated in this diagram

4.The Step Function (Step 4 in diagram) downloads the JSON input file from a local S3 bucket.

5.The next steps will read the Catalog file and perform validation to ensure the Catalog has no errors or missing data required by the API. Note This will prevent any errors while loading the catalog early in the process. Once the validation is completed, the lambda function will return to the Step function to progress on the catalog upload process.

6.The Step function will call a Lambda to record the catalog upload request to track the process. This will allow you to record all catalog upload requests

7.The Lambda will record the catalog upload request to the DynamoDB

8.Once the data recording is complete, the Lambda will return to the Step Function . The Step Function will trigger a Lambda function to begin the process to invoke the Amazon Catalog API.

9.The invoke API Lambda function will acquire STS credentials from the AWS IAM role to authorize the operations to upload the catalog upload to the Amazon.

10.The Lambda function will record the upload attempt to a DynamoDB table . This will enable tracking each catalog upload process.

Note: The IAM role must be allow listed on the Amazon systems to authorize the calls to the Amazon APIs. Failure to allow list the IAM role will result in a 403 Error when calling the Amazon API.

11.The Lambda function will call the Amazon API and pass the payload For detailed instructions on authenticating and calling the Amazon API, please see full solution. Amazon API will perform validation logic on the incoming request and respond with an ingestion ID or an error message if validation fails.

Note: The ingestion ID is used later on to look up and match incoming messages. You are responsible for storing the ingestion ID after receive the API response from Amazon. For more details on the JSON message response, please consult the API specification. Amazon will perform a search for each item based on the SKU. If an item matches then it will update the stored values using the values passed by the API. If no match is found, then a new item is created.

12.The invocation Lambda will receive and process the response from the Amazon API. 13.The invocation Lambda will update the CatalogAPIDB record with the status of the upload.

Note: If the API call returned an error, then there are no more events. If the API call returns an Ingestion ID then the process continues.

14.Once the processing concludes, Amazon will post a message to an SNS topic on the Amazon side.

15.The SNS topic will post the message payload to SQS in your AWS account.

Note: Please ensure that you have completed the steps to subscribe to the SNS topic.It is recommended to setup an extra notification on the SQS such as email. The customer is responsible for setting up notification on the SQS to be notified when Amazon posts a message. For instructions on how to setup notification, please see here.

16.Once a new message is received into SQS then the SQS service will invoke a lambda function to process the incoming message. The Lambda function will extract the Message element from the JSON which contains a URL for the JSON response from Amazon for the ingestion job.

17.The Lambda function will acquire STS credentials from the AWS IAM role to process the catalog upload response starting with the analysis of the SQS Queue message.

18.The lambda will download the result JSON from the pre-signed S3 link which contains the item results and stop. The link will be valid for 60 minutes before expiring.

19.The lambda will call the S3 Link and open up the JSON and store the result into the S3.

20.The Lambda will update the database with the SQS message specifically the pre-signed S3 URL for upload results.

The customer will process the results of the catalog upload on their side and perform any follow actions and clean up.