Using ECS tasks on AWS Fargate to replace Lambda functions
In a recent project one of the requirements was time-lapse video generation on request. The function would take a folder of images and stitch them together to create one timelapse video. Our initial thought was, as in most projects we do, was to use AWS Lambda, however we quickly ran into an issue. Lambda has a 15 minute timeout (From inital tests it was quite probable that large videos would take longer than this to process).
An introduction to AWS ECS and Fargate
Therefore we started to explore other options for on demand servers on AWS, and ECS Fargate became the obvious choice. ECS (Elastic Container Service) allows you to run Docker containers quickly and easily on AWS servers, so all you need is a working docker image with your app bundled in and you can quickly deploy full applications without a lot of the deployment "faf"!
There are two versions of ECS cluster:
- EC2 instance - your docker containers deploy to that instance and you pay for the compute power of the EC2 instance.
- Fargate - a serverless version where containers are ran on AWS managed servers which you don't have to pay for - instead you pay for compute time.
(Figure 1 explanation)
- The user requests the creation of a video to our serverless Lambda API.
- Lambda creates a task on ECS Fargate, which will be responsible for the video generation with environment parameters for S3 location of images.
- The task starts the docker container, which on start trigger the video generation process to begin.
- ECS task downloads the images for the time-lapse from an S3 bucket.
- ECS task creates the time-lapse video using ffmpeg .
- Store the time-lapse video to an S3 bucket ready for download (We could also notify the user at this point the video is ready via an S3 event).
- ECS task stops itself.
- User downloads the video from the S3 bucket.
As you can see there are quite a few parts here but broken down like this each part becomes quite simple. As this article is focused on ECS as a Lambda alternative I won't go into depth on most of the above points but focus on how we are managing the ECS tasks. Keep an eye out on our insights page for an article on how we used ffmpeg for video generation though!
This really basic Lambda function handler will trigger an ECS task to run on a Fargate launch type. Using the amplify or serverless frameworks you could quickly hook this up to AWS gateway to trigger the video generation process and replaces all the hard coded values with request parameters.
At the end of the video generation process the below code would run on the ECS conatiner and is responsible for triggering itself to shut down.
One thing you may have noticed in the above is that we are using “ECSCONTAINERMETADATA_URI” environment variable. This is a really useful endpoint ECS Fargate instances automatically have which will provide you with all sorts of metadata about the task you are currently running on. For the full docs see here .
After we have the cluster name and task id it’s a really simple usage of the ECS SDK to stop the task from running.
So that’s the whole process of triggering ECS tasks to run serverless containers in a similar way to Lambda but there are a few things you really need to consider when using this approach:
1. What happens if the process fails? - Make sure all errors are caught and result in the
stopTask() being called
2. What happens if the stopTask() function fails or the stopTask never gets called? - Part of the metadata in the from the
ECS_CONTAINER_METADATA_URI is the task
startedAt time. Using a scheduled Lambda running say once an hour we could poll against the
ECS_CONTAINER_METADATA_URI endpoint and check that the task hasn’t been running for too long. For example for our video generation we know that if the ECS task has been running for 2 hours or more then something has gone wrong. The we can log or alert someone that something went wrong and shutdown the task avoiding massive ECS costs.