Thundra Supports AWS Lambda’s Increased Ephemeral Storage
SUBSCRIBE TO OUR BLOG
Get our new blogs delivered straight to your inbox.
THANKS FOR SIGNING UP!
We’ll make sure to share the best materials crafted for you!
2 minutes read
Thundra Supports AWS Lambda’s Increased Ephemeral Storage
Written by Oguzhan Ozdemir
Solutions Engineer @Thundra
AWS announced AWS Lambda’s increased ephemeral storage (/tmp) configurable up to 10GB. This update introduces a drastic increase in the storage limits up to 20x which enables a lot of businesses to deal with large files, data and media processing, machine learning inference, and financial analysis.
As Thundra, we didn’t want to miss the opportunity to become a launch partner for such improvement and decided to test it out ourselves. The purpose of this article is to showcase our simple test on AWS Lambda’s increased ephemeral storage (/tmp) configurable up to 10GB.
If you want to learn more about AWS Lambda’s increased ephemeral storage (/tmp), make sure to visit the announcement post.
We tested this important update by ourselves and started enjoying the freedom it brings.
In our test, we have an AWS Lambda function that processes large files that are uploaded to our Amazon S3 bucket. The Lambda function first downloads the file in the `/tmp` folder and uses another file, index, or database to populate a new file. Then it uploads the processed file to an S3 bucket.
This is a pretty common scenario. Some similar use cases can be easily seen in various sectors such as the Lambda function could be processing some telemetry data, or it could be processing a phone call record, or it could be processing some payment record.
However, with the current 512 MB storage limitation, running your storage dependent, heavy operations on AWS Lambda doesn’t seem optimal. In fact, this storage factor of AWS Lambda was one of the reasons why engineers choose to create non-serverless solutions.
Let’s dive into our example and try our file processing function under a relatively heavy load. We upload a file that’s roughly 350 MB and expect it to be processed. Of course, as you can guess, we’ve instrumented our function with Thundra APM itself. So, we can observe our function in more detail.
In this screen, we see three invocations that happened back-to-back after we upload our file and all of them failed. Let’s click one of them and see it in detail.
Here, at the top right, we can see that all three invocations were in fact retries. Let’s click the TAGS and scroll down to see the error stack.
It seems that our Lamba function is running out of storage when processing larger files. In the past, that was it. We couldn’t change the storage capacity of our functions and we had to find another way to process these files. Perhaps, split them into smaller chunks or stream them from somewhere else and process it that way.
Thankfully, this is all changing and now there is a new capability in AWS Lambda. With this new change, we can now configure their function’s ephemeral storage (/tmp) according to our needs, between 512 MB and 10 GB. 10 GB being the magic number; AWS Lambda now supports 10 GB container images, 10 function memory, and 10 GB of ephemeral storage.
To configure our Lambda function with larger ephemeral storage, we click on the Configuration tab under the General Configuration section. We see that besides the Memory configuration, there is a new configuration for Ephemeral storage, which will be set at 512 MB by default.
By editing this, we are able to configure the ephemeral storage from 512 MB to 10240 MB. Remember, any value below 512 or above 10240 will return an error.
Let’s set it to 10 GB for the sake of this announcement and click Save. Our Lambda function is now ready to consume the previous heavy load.
Let’s trigger our function with the same file and expect it to be processed.
And, there we go. It’s working perfectly.
Serverless is becoming the default building block of every application more and more in every organization. Every day; Amazon Web Services, the creator of Lambda, is taking the technology one step further. The recent update of Lambda brings 20x more capability and freedom to applications developers. It enables many sectors such as data and media processing, machine learning, financial analysis, etc. to do their jobs more efficiently using serverless technologies.
We, as Thundra, are proud to be an advanced technology partner of AWS; and believe every new update for Lambda makes a huge impact on businesses.
SHARE
×
SUBSCRIBE TO OUR BLOG
Get our new blogs delivered straight to your inbox.
THANKS FOR SIGNING UP!
We’ll make sure to share the best materials crafted for you!
2 minutes read
Thundra Supports AWS Lambda’s Increased Ephemeral Storage
Written by Oguzhan Ozdemir
Solutions Engineer @Thundra
AWS announced AWS Lambda’s increased ephemeral storage (/tmp) configurable up to 10GB. This update introduces a drastic increase in the storage limits up to 20x which enables a lot of businesses to deal with large files, data and media processing, machine learning inference, and financial analysis.
As Thundra, we didn’t want to miss the opportunity to become a launch partner for such improvement and decided to test it out ourselves. The purpose of this article is to showcase our simple test on AWS Lambda’s increased ephemeral storage (/tmp) configurable up to 10GB.
If you want to learn more about AWS Lambda’s increased ephemeral storage (/tmp), make sure to visit the announcement post.
We tested this important update by ourselves and started enjoying the freedom it brings.
In our test, we have an AWS Lambda function that processes large files that are uploaded to our Amazon S3 bucket. The Lambda function first downloads the file in the `/tmp` folder and uses another file, index, or database to populate a new file. Then it uploads the processed file to an S3 bucket.
This is a pretty common scenario. Some similar use cases can be easily seen in various sectors such as the Lambda function could be processing some telemetry data, or it could be processing a phone call record, or it could be processing some payment record.
However, with the current 512 MB storage limitation, running your storage dependent, heavy operations on AWS Lambda doesn’t seem optimal. In fact, this storage factor of AWS Lambda was one of the reasons why engineers choose to create non-serverless solutions.
Let’s dive into our example and try our file processing function under a relatively heavy load. We upload a file that’s roughly 350 MB and expect it to be processed. Of course, as you can guess, we’ve instrumented our function with Thundra APM itself. So, we can observe our function in more detail.
In this screen, we see three invocations that happened back-to-back after we upload our file and all of them failed. Let’s click one of them and see it in detail.
Here, at the top right, we can see that all three invocations were in fact retries. Let’s click the TAGS and scroll down to see the error stack.
It seems that our Lamba function is running out of storage when processing larger files. In the past, that was it. We couldn’t change the storage capacity of our functions and we had to find another way to process these files. Perhaps, split them into smaller chunks or stream them from somewhere else and process it that way.
Thankfully, this is all changing and now there is a new capability in AWS Lambda. With this new change, we can now configure their function’s ephemeral storage (/tmp) according to our needs, between 512 MB and 10 GB. 10 GB being the magic number; AWS Lambda now supports 10 GB container images, 10 function memory, and 10 GB of ephemeral storage.
To configure our Lambda function with larger ephemeral storage, we click on the Configuration tab under the General Configuration section. We see that besides the Memory configuration, there is a new configuration for Ephemeral storage, which will be set at 512 MB by default.
By editing this, we are able to configure the ephemeral storage from 512 MB to 10240 MB. Remember, any value below 512 or above 10240 will return an error.
Let’s set it to 10 GB for the sake of this announcement and click Save. Our Lambda function is now ready to consume the previous heavy load.
Let’s trigger our function with the same file and expect it to be processed.
And, there we go. It’s working perfectly.
Serverless is becoming the default building block of every application more and more in every organization. Every day; Amazon Web Services, the creator of Lambda, is taking the technology one step further. The recent update of Lambda brings 20x more capability and freedom to applications developers. It enables many sectors such as data and media processing, machine learning, financial analysis, etc. to do their jobs more efficiently using serverless technologies.
We, as Thundra, are proud to be an advanced technology partner of AWS; and believe every new update for Lambda makes a huge impact on businesses.
SHARE
AWS announced AWS Lambda’s increased ephemeral storage (/tmp) configurable up to 10GB. This update introduces a drastic increase in the storage limits up to 20x which enables a lot of businesses to deal with large files, data and media processing, machine learning inference, and financial analysis.
As Thundra, we didn’t want to miss the opportunity to become a launch partner for such improvement and decided to test it out ourselves. The purpose of this article is to showcase our simple test on AWS Lambda’s increased ephemeral storage (/tmp) configurable up to 10GB.
If you want to learn more about AWS Lambda’s increased ephemeral storage (/tmp), make sure to visit the announcement post.
We tested this important update by ourselves and started enjoying the freedom it brings.
In our test, we have an AWS Lambda function that processes large files that are uploaded to our Amazon S3 bucket. The Lambda function first downloads the file in the `/tmp` folder and uses another file, index, or database to populate a new file. Then it uploads the processed file to an S3 bucket.
This is a pretty common scenario. Some similar use cases can be easily seen in various sectors such as the Lambda function could be processing some telemetry data, or it could be processing a phone call record, or it could be processing some payment record.
However, with the current 512 MB storage limitation, running your storage dependent, heavy operations on AWS Lambda doesn’t seem optimal. In fact, this storage factor of AWS Lambda was one of the reasons why engineers choose to create non-serverless solutions.
Let’s dive into our example and try our file processing function under a relatively heavy load. We upload a file that’s roughly 350 MB and expect it to be processed. Of course, as you can guess, we’ve instrumented our function with Thundra APM itself. So, we can observe our function in more detail.
In this screen, we see three invocations that happened back-to-back after we upload our file and all of them failed. Let’s click one of them and see it in detail.
Here, at the top right, we can see that all three invocations were in fact retries. Let’s click the TAGS and scroll down to see the error stack.
It seems that our Lamba function is running out of storage when processing larger files. In the past, that was it. We couldn’t change the storage capacity of our functions and we had to find another way to process these files. Perhaps, split them into smaller chunks or stream them from somewhere else and process it that way.
Thankfully, this is all changing and now there is a new capability in AWS Lambda. With this new change, we can now configure their function’s ephemeral storage (/tmp) according to our needs, between 512 MB and 10 GB. 10 GB being the magic number; AWS Lambda now supports 10 GB container images, 10 function memory, and 10 GB of ephemeral storage.
To configure our Lambda function with larger ephemeral storage, we click on the Configuration tab under the General Configuration section. We see that besides the Memory configuration, there is a new configuration for Ephemeral storage, which will be set at 512 MB by default.
By editing this, we are able to configure the ephemeral storage from 512 MB to 10240 MB. Remember, any value below 512 or above 10240 will return an error.
Let’s set it to 10 GB for the sake of this announcement and click Save. Our Lambda function is now ready to consume the previous heavy load.
Let’s trigger our function with the same file and expect it to be processed.
And, there we go. It’s working perfectly.
Serverless is becoming the default building block of every application more and more in every organization. Every day; Amazon Web Services, the creator of Lambda, is taking the technology one step further. The recent update of Lambda brings 20x more capability and freedom to applications developers. It enables many sectors such as data and media processing, machine learning, financial analysis, etc. to do their jobs more efficiently using serverless technologies.
We, as Thundra, are proud to be an advanced technology partner of AWS; and believe every new update for Lambda makes a huge impact on businesses.
AWS announced AWS Lambda’s increased ephemeral storage (/tmp) configurable up to 10GB. This update introduces a drastic increase in the storage limits up to 20x which enables a lot of businesses to deal with large files, data and media processing, machine learning inference, and financial analysis.
As Thundra, we didn’t want to miss the opportunity to become a launch partner for such improvement and decided to test it out ourselves. The purpose of this article is to showcase our simple test on AWS Lambda’s increased ephemeral storage (/tmp) configurable up to 10GB.
If you want to learn more about AWS Lambda’s increased ephemeral storage (/tmp), make sure to visit the announcement post.
We tested this important update by ourselves and started enjoying the freedom it brings.
In our test, we have an AWS Lambda function that processes large files that are uploaded to our Amazon S3 bucket. The Lambda function first downloads the file in the `/tmp` folder and uses another file, index, or database to populate a new file. Then it uploads the processed file to an S3 bucket.
This is a pretty common scenario. Some similar use cases can be easily seen in various sectors such as the Lambda function could be processing some telemetry data, or it could be processing a phone call record, or it could be processing some payment record.
However, with the current 512 MB storage limitation, running your storage dependent, heavy operations on AWS Lambda doesn’t seem optimal. In fact, this storage factor of AWS Lambda was one of the reasons why engineers choose to create non-serverless solutions.
Let’s dive into our example and try our file processing function under a relatively heavy load. We upload a file that’s roughly 350 MB and expect it to be processed. Of course, as you can guess, we’ve instrumented our function with Thundra APM itself. So, we can observe our function in more detail.
In this screen, we see three invocations that happened back-to-back after we upload our file and all of them failed. Let’s click one of them and see it in detail.
Here, at the top right, we can see that all three invocations were in fact retries. Let’s click the TAGS and scroll down to see the error stack.
It seems that our Lamba function is running out of storage when processing larger files. In the past, that was it. We couldn’t change the storage capacity of our functions and we had to find another way to process these files. Perhaps, split them into smaller chunks or stream them from somewhere else and process it that way.
Thankfully, this is all changing and now there is a new capability in AWS Lambda. With this new change, we can now configure their function’s ephemeral storage (/tmp) according to our needs, between 512 MB and 10 GB. 10 GB being the magic number; AWS Lambda now supports 10 GB container images, 10 function memory, and 10 GB of ephemeral storage.
To configure our Lambda function with larger ephemeral storage, we click on the Configuration tab under the General Configuration section. We see that besides the Memory configuration, there is a new configuration for Ephemeral storage, which will be set at 512 MB by default.
By editing this, we are able to configure the ephemeral storage from 512 MB to 10240 MB. Remember, any value below 512 or above 10240 will return an error.
Let’s set it to 10 GB for the sake of this announcement and click Save. Our Lambda function is now ready to consume the previous heavy load.
Let’s trigger our function with the same file and expect it to be processed.
And, there we go. It’s working perfectly.
Serverless is becoming the default building block of every application more and more in every organization. Every day; Amazon Web Services, the creator of Lambda, is taking the technology one step further. The recent update of Lambda brings 20x more capability and freedom to applications developers. It enables many sectors such as data and media processing, machine learning, financial analysis, etc. to do their jobs more efficiently using serverless technologies.
We, as Thundra, are proud to be an advanced technology partner of AWS; and believe every new update for Lambda makes a huge impact on businesses.
*** This is a Security Bloggers Network syndicated blog from Thundra blog authored by Oguzhan Ozdemir. Read the original post at: https://blog.thundra.io/thundra-supports-aws-lambdas-increased-ephemeral-storage