Simpler Observability with Lambda Telemetry API
2 minutes read
Simpler Observability with Lambda Telemetry API
Written by Oguzhan Ozdemir
Solutions Engineer @Thundra
AWS constantly improves the overall performance and tooling around AWS Lambda. The recent addition to this trend is the Lambda Telemetry API which enables its users to have increased observability around their Lambda functions.
AWS Lambda automatically captures the runtime telemetry data and sends them to the services like Amazon CloudWatch and Amazon X-Ray. With Telemetry API, it also allows you to create your own extensions and subscribe to these events to further process them or collect them to a place of your choosing. In more detail, there are three different telemetry streams for extensions to subscribe. These are; Function logs, Extension logs, and Platform telemetry data which has metrics and traces that describe events in the execution environment.
As Thundra, we wanted to make a small contribution to this new feature and open-source a Lambda layer that exports this telemetry information to an S3 bucket of the user’s choosing. You can find the layer and further information in this repository: https://github.com/thundra-io/lambda-telemetry-api-exporter
Lambda Execution Environment
Before the Telemetry API was introduced, the Lambda execution environment looked like this:
You can learn more about the Lambda Extension API here on AWS documentation. With Telemetry API, the Lambda execution environment now looks like this. Extensions can use Telemetry API to receive telemetry streams from the Lambda platform.
You can learn more about the Telemetry API and its extension implementation from the AWS blog post.
Using Thundra Telemetry API Exporter
We decided to write a simple extension that exports the collected telemetry data to an Amazon Kinesis Data Firehose delivery stream. Once the data is delivered to the Firehose stream, the destination is up to the users. You can deliver it directly to an S3 bucket, to a database, or you can handle it by yourself.
The extension is still in development and there is certainly room for improvement. However, it’s still a good proof-of-concept work that shows what you can do with the new Telemetry API. The extension layer currently supports NodeJS runtimes and is available in almost all regions.
To learn more about the exporter, you can visit the public GitHub repository. As to show how it works, the requirements are simple. First, you need to have your Firehose Delivery Stream ready. If you don’t know what Firehose is, you can visit the AWS Documentation page here.
Once that’s ready you can add the below layer to your lambda by replacing the {region} with yours:
arn:aws:lambda:{region}:269863060030:layer:lambda-telemetry-api-exporter:1The next step would be adding permissions and setting the environment variable that’s required by the Layer. Your Lambda function should have access to the Firehose stream, so a simple permission policy would look like the following. Make sure to replace the delivery stream ARN with the correct one.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"firehose:DeleteDeliveryStream",
"firehose:PutRecord",
"firehose:PutRecordBatch",
"firehose:UpdateDestination"
],
"Resource": [
// Replace this with yours.
"arn:aws:firehose:region:account-id:deliverystream/delivery-stream-name"
]
}
]
}
To learn more about the advanced access controls, visit the documentation page.
Then, set the environment variable DELIVERY_STREAM_NAME to your stream name. Once all this is set correctly, the data from the Telemetry API should flow to your destination for further processing. For this example, we’ve set up the destination to an S3 bucket.
If we open one of them, we can see what sort of data that’s flowing through our pipeline.
Great! There are endless possibilities at this point. You can filter the data, process it and query it as your own need. This kind of capability is what excites us and it’s nice to showcase one of them here.
Conclusion
We can see the trend of observability taking over our everyday environment and we’re more than happy to be a part of it, especially when these capabilities are coming to our serverless toolsets. The constant effort of the engineers at AWS to increase the potential of serverless is profound.
SHARE
×
SUBSCRIBE TO OUR BLOG
Get our new blogs delivered straight to your inbox.
THANKS FOR SIGNING UP!
We’ll make sure to share the best materials crafted for you!
2 minutes read
Simpler Observability with Lambda Telemetry API
Written by Oguzhan Ozdemir
Solutions Engineer @Thundra
AWS constantly improves the overall performance and tooling around AWS Lambda. The recent addition to this trend is the Lambda Telemetry API which enables its users to have increased observability around their Lambda functions.
AWS Lambda automatically captures the runtime telemetry data and sends them to the services like Amazon CloudWatch and Amazon X-Ray. With Telemetry API, it also allows you to create your own extensions and subscribe to these events to further process them or collect them to a place of your choosing. In more detail, there are three different telemetry streams for extensions to subscribe. These are; Function logs, Extension logs, and Platform telemetry data which has metrics and traces that describe events in the execution environment.
As Thundra, we wanted to make a small contribution to this new feature and open-source a Lambda layer that exports this telemetry information to an S3 bucket of the user’s choosing. You can find the layer and further information in this repository: https://github.com/thundra-io/lambda-telemetry-api-exporter
Lambda Execution Environment
Before the Telemetry API was introduced, the Lambda execution environment looked like this:
You can learn more about the Lambda Extension API here on AWS documentation. With Telemetry API, the Lambda execution environment now looks like this. Extensions can use Telemetry API to receive telemetry streams from the Lambda platform.
You can learn more about the Telemetry API and its extension implementation from the AWS blog post.
Using Thundra Telemetry API Exporter
We decided to write a simple extension that exports the collected telemetry data to an Amazon Kinesis Data Firehose delivery stream. Once the data is delivered to the Firehose stream, the destination is up to the users. You can deliver it directly to an S3 bucket, to a database, or you can handle it by yourself.
The extension is still in development and there is certainly room for improvement. However, it’s still a good proof-of-concept work that shows what you can do with the new Telemetry API. The extension layer currently supports NodeJS runtimes and is available in almost all regions.
To learn more about the exporter, you can visit the public GitHub repository. As to show how it works, the requirements are simple. First, you need to have your Firehose Delivery Stream ready. If you don’t know what Firehose is, you can visit the AWS Documentation page here.
Once that’s ready you can add the below layer to your lambda by replacing the {region} with yours:
arn:aws:lambda:{region}:269863060030:layer:lambda-telemetry-api-exporter:1The next step would be adding permissions and setting the environment variable that’s required by the Layer. Your Lambda function should have access to the Firehose stream, so a simple permission policy would look like the following. Make sure to replace the delivery stream ARN with the correct one.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"firehose:DeleteDeliveryStream",
"firehose:PutRecord",
"firehose:PutRecordBatch",
"firehose:UpdateDestination"
],
"Resource": [
// Replace this with yours.
"arn:aws:firehose:region:account-id:deliverystream/delivery-stream-name"
]
}
]
}
To learn more about the advanced access controls, visit the documentation page.
Then, set the environment variable DELIVERY_STREAM_NAME to your stream name. Once all this is set correctly, the data from the Telemetry API should flow to your destination for further processing. For this example, we’ve set up the destination to an S3 bucket.
If we open one of them, we can see what sort of data that’s flowing through our pipeline.
Great! There are endless possibilities at this point. You can filter the data, process it and query it as your own need. This kind of capability is what excites us and it’s nice to showcase one of them here.
Conclusion
We can see the trend of observability taking over our everyday environment and we’re more than happy to be a part of it, especially when these capabilities are coming to our serverless toolsets. The constant effort of the engineers at AWS to increase the potential of serverless is profound.
SHARE
AWS constantly improves the overall performance and tooling around AWS Lambda. The recent addition to this trend is the Lambda Telemetry API which enables its users to have increased observability around their Lambda functions.
AWS Lambda automatically captures the runtime telemetry data and sends them to the services like Amazon CloudWatch and Amazon X-Ray. With Telemetry API, it also allows you to create your own extensions and subscribe to these events to further process them or collect them to a place of your choosing. In more detail, there are three different telemetry streams for extensions to subscribe. These are; Function logs, Extension logs, and Platform telemetry data which has metrics and traces that describe events in the execution environment.
As Thundra, we wanted to make a small contribution to this new feature and open-source a Lambda layer that exports this telemetry information to an S3 bucket of the user’s choosing. You can find the layer and further information in this repository: https://github.com/thundra-io/lambda-telemetry-api-exporter
Lambda Execution Environment
Before the Telemetry API was introduced, the Lambda execution environment looked like this:
You can learn more about the Lambda Extension API here on AWS documentation. With Telemetry API, the Lambda execution environment now looks like this. Extensions can use Telemetry API to receive telemetry streams from the Lambda platform.
You can learn more about the Telemetry API and its extension implementation from the AWS blog post.
Using Thundra Telemetry API Exporter
We decided to write a simple extension that exports the collected telemetry data to an Amazon Kinesis Data Firehose delivery stream. Once the data is delivered to the Firehose stream, the destination is up to the users. You can deliver it directly to an S3 bucket, to a database, or you can handle it by yourself.
The extension is still in development and there is certainly room for improvement. However, it’s still a good proof-of-concept work that shows what you can do with the new Telemetry API. The extension layer currently supports NodeJS runtimes and is available in almost all regions.
To learn more about the exporter, you can visit the public GitHub repository. As to show how it works, the requirements are simple. First, you need to have your Firehose Delivery Stream ready. If you don’t know what Firehose is, you can visit the AWS Documentation page here.
Once that’s ready you can add the below layer to your lambda by replacing the {region} with yours:
arn:aws:lambda:{region}:269863060030:layer:lambda-telemetry-api-exporter:1The next step would be adding permissions and setting the environment variable that’s required by the Layer. Your Lambda function should have access to the Firehose stream, so a simple permission policy would look like the following. Make sure to replace the delivery stream ARN with the correct one.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"firehose:DeleteDeliveryStream",
"firehose:PutRecord",
"firehose:PutRecordBatch",
"firehose:UpdateDestination"
],
"Resource": [
// Replace this with yours.
"arn:aws:firehose:region:account-id:deliverystream/delivery-stream-name"
]
}
]
}
To learn more about the advanced access controls, visit the documentation page.
Then, set the environment variable DELIVERY_STREAM_NAME to your stream name. Once all this is set correctly, the data from the Telemetry API should flow to your destination for further processing. For this example, we’ve set up the destination to an S3 bucket.
If we open one of them, we can see what sort of data that’s flowing through our pipeline.
Great! There are endless possibilities at this point. You can filter the data, process it and query it as your own need. This kind of capability is what excites us and it’s nice to showcase one of them here.
Conclusion
We can see the trend of observability taking over our everyday environment and we’re more than happy to be a part of it, especially when these capabilities are coming to our serverless toolsets. The constant effort of the engineers at AWS to increase the potential of serverless is profound.
AWS constantly improves the overall performance and tooling around AWS Lambda. The recent addition to this trend is the Lambda Telemetry API which enables its users to have increased observability around their Lambda functions.
AWS Lambda automatically captures the runtime telemetry data and sends them to the services like Amazon CloudWatch and Amazon X-Ray. With Telemetry API, it also allows you to create your own extensions and subscribe to these events to further process them or collect them to a place of your choosing. In more detail, there are three different telemetry streams for extensions to subscribe. These are; Function logs, Extension logs, and Platform telemetry data which has metrics and traces that describe events in the execution environment.
As Thundra, we wanted to make a small contribution to this new feature and open-source a Lambda layer that exports this telemetry information to an S3 bucket of the user’s choosing. You can find the layer and further information in this repository: https://github.com/thundra-io/lambda-telemetry-api-exporter
Lambda Execution Environment
Before the Telemetry API was introduced, the Lambda execution environment looked like this:
You can learn more about the Lambda Extension API here on AWS documentation. With Telemetry API, the Lambda execution environment now looks like this. Extensions can use Telemetry API to receive telemetry streams from the Lambda platform.
You can learn more about the Telemetry API and its extension implementation from the AWS blog post.
Using Thundra Telemetry API Exporter
We decided to write a simple extension that exports the collected telemetry data to an Amazon Kinesis Data Firehose delivery stream. Once the data is delivered to the Firehose stream, the destination is up to the users. You can deliver it directly to an S3 bucket, to a database, or you can handle it by yourself.
The extension is still in development and there is certainly room for improvement. However, it’s still a good proof-of-concept work that shows what you can do with the new Telemetry API. The extension layer currently supports NodeJS runtimes and is available in almost all regions.
To learn more about the exporter, you can visit the public GitHub repository. As to show how it works, the requirements are simple. First, you need to have your Firehose Delivery Stream ready. If you don’t know what Firehose is, you can visit the AWS Documentation page here.
Once that’s ready you can add the below layer to your lambda by replacing the {region} with yours:
arn:aws:lambda:{region}:269863060030:layer:lambda-telemetry-api-exporter:1The next step would be adding permissions and setting the environment variable that’s required by the Layer. Your Lambda function should have access to the Firehose stream, so a simple permission policy would look like the following. Make sure to replace the delivery stream ARN with the correct one.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"firehose:DeleteDeliveryStream",
"firehose:PutRecord",
"firehose:PutRecordBatch",
"firehose:UpdateDestination"
],
"Resource": [
// Replace this with yours.
"arn:aws:firehose:region:account-id:deliverystream/delivery-stream-name"
]
}
]
}
To learn more about the advanced access controls, visit the documentation page.
Then, set the environment variable DELIVERY_STREAM_NAME to your stream name. Once all this is set correctly, the data from the Telemetry API should flow to your destination for further processing. For this example, we’ve set up the destination to an S3 bucket.
If we open one of them, we can see what sort of data that’s flowing through our pipeline.
Great! There are endless possibilities at this point. You can filter the data, process it and query it as your own need. This kind of capability is what excites us and it’s nice to showcase one of them here.
Conclusion
We can see the trend of observability taking over our everyday environment and we’re more than happy to be a part of it, especially when these capabilities are coming to our serverless toolsets. The constant effort of the engineers at AWS to increase the potential of serverless is profound.
*** This is a Security Bloggers Network syndicated blog from Thundra blog authored by Oguzhan Ozdemir. Read the original post at: https://blog.thundra.io/simpler-observability-with-lambda-telemetry-api
