7 ways to access your organizational backbone (powered by AWS)

Engaging in a cloud engineer's mindset encourages many responsibilities. The cloud is a great place for developers and has been proven to scale better than most on-prem deployments. The sheer amount of available services to a developer is something I love about AWS. However, engineers often do not fulfill their responsibilities or get sloppy during the implementation of an architecture. This ultimately leads to an insecure environment. This post will focus on 7 pitfalls that most organizations overlook when migrating into the cloud. It shows the relevance of being updated about the latest launches of AWS features. Most of the problems may not apply to you when best practices in terms of privilege management are in place.

The following examples assume that developers have administrative access to AWS accounts. It should illustrate what´s possible to configure when an IT infrastructure department delegates too many privileges to developer teams. It also assumes that the developer team isn´t used to working on AWS and wants to make "fast" progress by ignoring best practices.

1. VPC Lattice

VPC Lattice announced GA on 31.3.2023 (see also Simplify Service-to-Service Connectivity, Security, and Monitoring with Amazon VPC Lattice – Now Generally Available | AWS News Blog). VPC Lattice can utilize AWS Private Link to connect services running in AWS in a modern way. It allows developers to interconnect distributed components with layer 7 routing functionality. The service is designed to be consumed by 2 Roles. A network administrator is responsible for managing service networks and a service owner can manage service endpoints and the connectivity to available service networks. This can be seen in the following picture:

Many organizations have a very generous approach to granting AWS access to developers. And this makes sense: Nowadays, speed matters and you cannot always wait for third-party teams during prototyping or PoCs. Let's assume the developer team is new to AWS and has admin privileges in a test account. What could happen is that a VPC Lattice service network connected to an internal VPC gets provisioned (2) without informing the team responsible for networking. The result: You may have introduced a backdoor (3) into your organization's network which is not secured by your corporate guidelines. Your internal service may get exposed to a publicly available service (1) that allows attackers to reach internal resources without passing any corporate security gates (4).

2. VPC Endpoint Service

VPC Endpoint and Endpoint Services are great when you want to build connectivity to services hosted in the cloud. In the backend, this is possible through AWS Private Link (aka the AWS Network backbone) and Route53 (Private Zone). Three common examples are:

1. AWS Service connectivity: It allows you to connect your VPC with AWS services like a secrets manager or SSM Parameter Store without having any internet connection.

2. Publish Services owned by you: It allows you to provide a private endpoint for a service you own. This is possible via different integrations. One example is to expose your internal network load balancer (NLB) via AWS private link. The consumer can deploy a VPC endpoint to your service endpoint that allows direct backend connectivity between the consumer's VPC and your published service. Again: No internet access is required.

3. Implement (1) or (2) for your on-premise network: Many companies are making use of DNS Alias and redirect or "tunnel" traffic through a VPC Endpoint available inside your corporate network backbone.
   

Since a VPC Endpoint service is deployed inside your VPC, this introduces a new network path into your corporate backbone. Here we face the same issue as with VPC Lattice. If a developer has the privilege to configure VPC Endpoint services, chances are high that the provisioned connection will run without any attention from a network or security team. This may bypass internal corporate security gates.

3. Cloudshell

Ever required the AWS API when working on the AWS console? Cloud shell is a solution that allows you to run a small Amazon Linux instance to run basic commands within a terminal (shell). Since some information may not be exposed via AWS Console or you need to quickly run a small script (for example a cli query with a pipe to jq to extract relevant information) this little helper saves you precious time.

On June 26, 2024, AWS announced VPC support for CloudShell (see also: AWS CloudShell now supports Amazon Virtual Private Cloud (VPC) - AWS). This means that everyone who can start a cloud shell session will have an available compute instance inside a VPC. This is unacceptable for an IT organization as the instance is not managed and governed by the organization's IT department. The cloud shell instance can be used by an attacker as an entry path into a corporate network. All that is needed is to hijack one AWS session.

4. EC2 Instance Connect

EC2 Instance connect allows you to access EC2 Machine via an active AWS session. It was launched on 14.06.2023 (see also: Secure Connectivity from Public to Private: Introducing EC2 Instance Connect Endpoint | AWS Compute Blog (amazon.com)).

I used the service myself when I had to set up EC2 machines in a private subnet which was not connected to any backbone. In my case, the affected machine was an SD-WAN router which didn´t come up. Instance Connect helped me to gain access through the private IP of the router which allowed me to troubleshoot the problem. While this comes in handy it also introduces a new backdoor into your corporate network. All that is needed is to have privileges to create the instance connect endpoint and an active AWS session. Hopefully, you have either a secured login or a key deployed for your EC2 machines. Otherwise, the attacker can get administrative privileges as a local admin. Also if the user is allowed to spin up new EC2 machines with unverified AMI it is easy to gain access from a public network into the private VPC.

5. IAM:PassRole

AWS offers many services that require execution roles. Let´s take AWS Lambda as an example: Whenever a Lambda function is executed it will inherit AWS permission via a service-linked role. This means that the execution of the Lambda function assumes an IAM Role which has a configured trust policy with the lambda service. With IAM:PassRole a user can determine which available execution roles can be assigned to a lambda function. This allows for example to pass the role "aws-controltower-ForwardSnsNotificationRole" to a newly created lambda function. As a consequence, other privileges than intended are granted for the lambda function. This generates security issues for example with another service called codebuild. AWS codebuild often is one part of a CICD pipeline. Let us assume one AWS account was used to host different CICD Pipelines for different products. Each pipeline has privileges to connect to different AWS accounts to provision resources (IaC). If an AWS user is able to create new code-build projects IAM:PassRole is used to inherit privileges. Who says I cannot use the same execution role as used by another codebuild project? This would have enabled an attacker who has access to a developer account to break into a productive account and expose one of the other problems mentioned in this post.

6. SSM Manager

Amazon Systems Manager is a great service to operate and automate compute resources. In my opinion, it is rather a whole suite of services as it allows you to do so much more: Instance fleet management, patch management, automation/configuration management, and more. SSM Manager also gets used to easily connecting to an EC2 machine. All you have to do is to install an agent on the machine and let the agent connect to the service. AWS sells this connection as a more secure way to connect than using ssh or rdp. I partially agree with this statement. Why only partially? It may introduce a way to connect to internal resources via public transport and bypass corporate logins. By relying solely on a AWS session a user may gain administrative privileges without using any kind of domain login. (see also: Connecting to a Windows Server managed instance using Remote Desktop - AWS Systems Manager (amazon.com)).

7. Public Services: API Gateway (Example), Sagemaker

Many services are using ENIs to gain access to a VPC. I have prepared two examples to illustrate this problem. Example 1 - API Gateway and Lambda functions: A common way for organizations to implement internet access is to deny the attachment of internet gateways (IGWs) to VPCs via service control policies (SCPs) and configure a central internet breakout behind a transit gateway. However, just because a user cannot directly attach an internet gateway doesn´t necessarily mean there is no solution to connect the internet with my local network. A developer may deploy a lambda function inside a VPC and make it invocable through the internet over the API Gateway service. This is another example of exposing internal services to the outside and bypassing potential internal security gates using AWS services instead of corporate-provided solutions. In case of problems in the code of the lambda function, an attacker may even be able to hijack the sessions of the lambda function itself. And now imagine you´ve given your lambda function admin privileges to be faster in your product delivery...

Example 2 - Sagemaker Studio: AWS Sagemaker is a service for data engineers. It allows you to use common tools that are needed when you build data products or machine learning (ML) solutions. The Sagemaker studio allows you to create an instance that has direct internet access on the one side and VPC access through an exposed ENI on the other side. The host itself is not governed by any IT department and a user is free to access the internet via an Amazon-provided VPC - without passing any internal security gates. (See also: Connect Amazon SageMaker Studio in a VPC to External Resources - Amazon SageMaker)

Lessons learned

This post proves that engineering on AWS needs responsible engineers and a strong team governing AWS itself. Even though a developer may get access to a variety of services, he or she should always think carefully about the consequences of implemented solutions. Also, the team responsible for governing AWS must keep itself updated - at all times. You never know when AWS will introduce the next feature which may allow users to dig a new hole into your corporate network backbone. In general AWS and you as an engineer are doing a great job. A cloud mindset is often misunderstood by different entities in different positions in an enterprise. I hope you have enjoyed reading this blog post and sharpened your sense of security - if you feel you forgot to think about any point above don´t be shy and escalate the matter to the management or try to solve it yourself if you are responsible.