Securing Beats at Scale
At Hysolate we use Elastic to collect anonymous telemetry events. The Elastic stack is a great set of tools for centralized monitoring and log collection. However, the actual data shippers, called “Beats”, are not optimized for managing authentication at scale.
In this post, I will present how we overcame this issue with a modern – and very clever – security and authentication mechanism.
The Problem
Hysolate sends anonymous telemetry events to its cloud infrastructure to show our customers their usage information and to let them audit corporate access. For example, customers can get alerts on blocked operations, such as attempted transfers from Workspace.
As a cybersecurity-focused company, we require our telemetry service to use the most up-to-date industry-standard security practices.
For secure and scalable telemetry events delivery, the following requirements must be met:
- The transport layer itself must be secure and encrypted
- The receiving end must be able to authenticate the sender in a secure manner, at scale
- The receiving end must be able to withstand (D)DoS attacks
A typical Elastic stack server deployment uses:
- Elasticsearch, for storing the collected telemetry events; and –
- Logstash, as an additional preprocessing pipeline.
Elasticsearch and Logstash provide the following solutions for some of the requirements above:
- Elasticsearch and Logstash support TLS as a server.
- Elasticsearch supports multiple authentication options: TLS client certificates, API key, and user credentials (basic authentication). Logstash supports only TLS client certificates with the Lumberjack protocol. Unfortunately, there is no easy way to rotate and distribute them to all devices that are sending telemetry data.
- Elasticsearch and Logstash do not provide anti-(D)DoS protection for unauthenticated requests.
Enjoying the Benefits of Beats for Telemetry while Staying Secure
To send such telemetry data from devices into Elasticsearch, it is common to use Beats – a group of client applications that send them to Logstash or Elasticsearch. For best performance, Beats uses the Lumberjack protocol to send telemetry events to Logstash.
However, Lumberjack is not a HTTP-based protocol, and therefore cannot be protected by HTTP-based security solutions.
Beats can also use a SOCKv5 proxy with optional basic authentication. But as with Elasticsearch, basic authentication is not easy to rotate, and many organizations configure their on-premise firewalls to block such traffic.
So we were stuck: how do we enjoy the benefits of Beats for telemetry while making sure it’s secure?
Hysolate’s Elegant Solution
Luckily, we found an elegant solution that is both secure and efficient. Before we dive into it, here’s a bit of additional background: Hysolate’s cloud infrastructure is deployed behind a HTTPS CDN, and utilizes Signed JSON Web Tokens as the authentication method both for Console users as well as internal API of the software on the devices.
Hysolate’s cloud infrastructure validates the Signed JSON Web Token on each HTTP request at the CDN Edge servers, so only authenticated requests reach Hysolate’s backend systems. This approach allows Hysolate’s cloud infrastructure to withstand unauthenticated requests without wasting resources.
To achieve both performance and scalability of the delivery of telemetry events, we combined the best of both worlds by wrapping the Lumberjack protocol with WebSocketSecure.
WebSocketSecure is HTTP-based, and therefore gains the security features of CDN and Signed JSON Web Tokens.
At the cloud-infrastructure level, we deployed an internal auto-scaling group of WebSocketSecure-to-Lumberjack unwrapper backend services. These services forward the Lumberjack traffic to an internal auto-scaling group of Logstash instances, which process the telemetry events, and forward to the internal Elasticsearch cluster.
On devices with Hysolate, we install a wrapper service alongside the Beats services. The wrapper exposes itself to the Beats services as a local Logstash, and it wraps the Lumberjack protocol with WebSocketSecure. The wrapper uses the same authentication method as the one used by our own internal API.
This approach allows us to provide our customers with the benefits of visibility into their devices – without compromising on security.
The post Securing Beats at Scale appeared first on Hysolate.
*** This is a Security Bloggers Network syndicated blog from Blog – Hysolate authored by Boris Figivsky. Read the original post at: https://www.hysolate.com/blog/securing-beats-at-scale/
