Home Artificial Intelligence How I Built A Cascading Data Pipeline Based on AWS (Part 2)

How I Built A Cascading Data Pipeline Based on AWS (Part 2)

1
How I Built A Cascading Data Pipeline Based on AWS (Part 2)

Automatic, scalable, and powerful

Photo by Mehmet Ali Peker on Unsplash

Previously, I shared my experience in developing a knowledge pipeline using AWS CloudFormation technology. It will not be an optimal approach, though, since it leaves behind 3 more issues awaiting resolution:

  1. The deployment must be imposed manually which could increase the possibilities of errors;
  2. All resources are created in a single single stack, without proper boundaries and layers; as the event cycle goes on, the resource stack shall be heavier, and managing it would be a disaster;
  3. Many resources are alleged to be sustained and reused in other projects.

Briefly, we’re going to increase the manageability and reusability of this project, in an agile manner.

AWS enables users to implement 2 forms of CloudFormation structural patterns: cross-stack reference and nested stacking. Cross-stack reference stands for a designing form of developing cloud stacks individually, and typically independently, while the resources amongst all stacks may be interrelated based on the reference relationship. Nested stacking means a CloudFormation stack composed of other stacks. It’s achieved through the use of the AWS::CloudFormation::Stack resource.

A nested stack in real life: a nest stuffed with nests/eggs (Photo by Giorgi Iremadze on Unsplash)

Because certainly one of our missions we aim to realize is to give you higher project management, the project goes to be broken down by layered separation and nested stacking is the one to assist. Nonetheless, in regard to the intrinsic interrelationship between the artifacts of the present stack, we’d also have to take a drop of cross-stack reference.

We created 3 Lambda functions, 3 DynamoDB tables, 1 IAM role together with its policies attached, several SQS queues, and several other Cloudwatch alarms. Attributable to the complexity of the functions themselves, on this version, they will be defined in separate templates, with the services only utilized by themselves including alarms and dead letter queues. Aside from those, IAM resources shall be…

1 COMMENT

LEAVE A REPLY

Please enter your comment!
Please enter your name here