Time to Read: 7 min. 49 sec.
tl;dr: If you're wondering about network decryption in the cloud -- specifically using Amazon's new VPC traffic mirroring, then read our responses to six critical questions following a webinar we recently held on the subject.
We recently hosted a Webinar with Amazon AWS called, “Gaining Decrypted Visibility in the Public Cloud With Amazon VPC Traffic Mirroring” (click to watch the webinar on demand).
Speaking from Amazon was Anoop Dawani and speaking from Nubeva was Steve Perkins. Erik Freeland from Nubeva was the live demo guide for the webinar.
The presentation spurred a number of questions from the audience that deserved more complete answers than time allowed on the webinar, so we’re elaborating on them here. If you’d like to view/listen to the presentation for yourself, please do so here.
Question 1: Can you provide more details on the performance and scaling of the solution?
Answer: Amazon VPC traffic mirroring / packet capture runs at line speed. Because it is an infrastructure-based solution provided by Amazon, there is no overhead to the CPU. That means there’s amazing performance. Because mirrors are set up as part of the AWS environment, they scale along with your environment. Mirrors will respond to whatever tools you use to orchestrate your environment whether that’s Chef, Puppet, Ansible, fancy AWS lambdas or even some guy in IT who writes scripts by hand.
On Nubeva’s TLS decrypt side, the key agent has been tested to 10,000 key extractions per minute - the testing, not the technology, threshold. If you run these in parallel, you can get to millions of extractions per minute. The agents run in a sub-10mb memory footprint and consume less than 2% utilization of a single core of a CPU. It’s an incredibly lightweight architecture for a nearly invisible, read-only probe.
The symmetric key database is a customer’s database. It’s not Nubeva’s database. Today we use Amazon DynamoDB which is an incredibly secure, fast, flexible and near-infinitely scalable NoSQL database. It is so secure and compliant it is used in the banking and finance industry. It is so performant that it is used in the gaming and retail industries.
The decryptor agent is also very scalable and fast. We have clocked it on a C5.X-large doing north of 2.1Gbps with minimal CPU usage and that is even supporting modern ciphers like ChaCha20.
The whole solution is highly performant and scales in and out (horizontally) as well as up and down. Because each component scales independently, massively scalable outcomes are possible. We have tested parallel decryption in the TBps throughput on decryption (when running in parallel).
Question 2: In a Kubernetes environment, does your agent run on the node (VM) or on the pod (container)? Can it extract keys from pods that come and go (microservices)?
Answer: The Nubeva key discovery agent runs in Kubernetes as a DaemonSet. A DaemonSet ensures that all Kubernetes Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. The Nubeva agent can be thought of as a special kind of container installed in the host system as a DaemonSet. There will only ever be one Nubeva key discovery agent per Node this way. Whenever any kind of container or microservice comes up in that Node, the Nubeva agent is able to discover and extract the symmetric keys generated in each TLS handshake that the containers in the Kubernetes cluster may be participating in.
Whichever way Kubernetes decides to host all of its containers on a Node, the Nubeva agent can monitor all of them regardless if there are 40 containers or 4 and regardless of how long the container exists.
Question 3: How easy or difficult would it be for external actors to scan memory to intercept the key from the TLS handshake? In other words, what is stopping a threat actor from gathering enough intelligence to insert their own agent onto a VM or into a container environment?
Answer: This question misses the boat. It is one thing for the owner of a system to have access to its memory. It is a very different matter for an unauthorized person to break in and access system memory. The question is similar to saying, “I see that you can check your own bank account and see how much money is in there. But what is stopping a bank robber from coming in and stealing your money?” The two scenarios are as different as apples and Saturn’s rings.
It is very hard for an external, bad actor to get access to core internal resources. First, they would need system resources access to do so. This alone should be quite difficult given security groups, IAM rules, and basic security.
But, let’s assume for the sake of argument, that they do get access to these resources. They would have to find where in memory the symmetric keys exist. Remember that the keys are not stored, they’re ephemeral. Nubeva uses machine learning to discover the symmetric key locations in memory.
But, let’s assume for the sake of argument, that they do get access to the memory spaces where the symmetric keys are located at the time the keys exist there. At this point, it’s probably not worth it for the hacker. After all, these are just the ephemeral symmetric keys. Each symmetric key only encrypts/decrypts the current session. So the hacker has to also get a pcap of the current session and the symmetric key. And then, to get anything of value, they have to do the same thing, over and over, for each and every session and each and every symmetric key. This is why perfect forward secrecy and TLS 1.3 are so much stronger than prior standards and practices. There are no more “master keys to the kingdom” that will decrypt everything with one key.
The reality is that, if a hacker has penetrated your application so far that they can read all memory at all those times, they might as well just read the pre-encrypted data in the application rather than trying to bother with the traffic intercept.
If a hacker or other bad actor has enough permission to put some "code" on a box, it is 100% compromised regardless of the TLS handshake.
Remember that PFS / TLS1.3 protects the communication of two entities. It’s not meant to protect the entities themselves. That’s why one needs forensics to see if there are hackers at the endpoints.
Question 4: How does the tool interface with the decryption process? Are there any limitations to what tools you can decrypt on?
Answer: In order to provide an end-to-end solution, Nubeva has created a universal decryptor that runs, as a container, on the tool instance itself. Currently, the Nubeva decryptor runs on Linux instances and supports any tool that has a Linux-based OS. This includes many open source tools such as Moloch, Suricata, Zeek, and others that perform deep packet inspection, detection, and DLP functions. It also includes a very large set of commercial tools such as Protectwise (now Verizon Network Detection and Response), RSA NetWitness, Symantec Secure Analytics (now owned by Broadcom), and more.
The decryptor itself listens on eth0 and looks for any encrypted traffic, once it sees encrypted traffic, it buffers the stream while it matches up the session key from the secure key database, and then generates a new session with the decrypted payload to be replayed on the tool. The original, encrypted traffic stream is also passed through with all security, session and encryption headers intact. This way, your tools have access to both the decrypted and originally encrypted traffic available for analysis, detection, and processing.
Question 5: How does the key agent find the keys and where are they installed?
Answer: Obtaining final encryption keys from certificates or master keys and libraries is impossible with the new TLS 1.3 standard.
Nubeva has created a new architecture for decryption called symmetric key intercept.
Nubeva’s Key Discovery Agent discovers the final, ephemeral, symmetric keys for the TLS session from the working memory of either the TLS client or TLS host. This includes native applications and containers running on that host. Discovery happens in real-time after the TLS handshake without application modification, library modifications, shims or debug mode requirements. The agent needs to be on only one side of the TLS handshake to work. This opens the door for full, out-of-band decryption of traffic from sites and APIs using pinned certificates, perfect forward secrecy, or TLS 1.3.
Key discovery happens because the Nubeva agent uses AI-based memory location signatures to find and identify the symmetric keys in working memory. Nubeva uses machine-learning and AI-based rules to create the signatures. Agents can pull signatures to them from Nubeva’s SaaS management console (signatures are never pushed) or from a company’s own, internal secure servers.
The Nubeva Key Discovery Agent can extract over 10,000 keys per minute at less than 1% CPU.
This method of obtaining the final, ephemeral, symmetric keys from memory immediately after the TLS handshake breathes new life and value into established workflows, tools and teams.
This symmetric key intercept approach avoids decommissioning out-of-band infrastructure, does not require that you buy or lock-in to new tools and gives full visibility for even the newest decentralized and distributed cloud environments.
Question 6: Does decryption work with pinned certificates?
Answer: Yes! Session key extraction works with virtually any encryption -- there is no limitation tied to certificates and private keys, including pinned certificates. The session keys are discovered and extracted from memory from either side of the conversation--client or server-side.
Once again, we encourage you to watch the webinar on demand to get all of your questions answered: https://info.nubeva.com/webinar_nubeva_tls_aws_traffic_mirroring