Harden your Serverless API (Monitoring and Encryption)
In my previous post, we went over the basics of hardening the security of serverless functions. We highlighted the importance of good practices such as input validation, proper authentication, and fine-grained authorization. In this post, we will take a look at an additional set of best practices related to proper monitoring and logging, third-party dependency management, and sensitive information encryption.
Common Vulnerabilities And Their Mitigation (continued)
The next three items in the 12 Most Critical Risks for Serverless Applications published by the Cloud Security Alliance have to do with vulnerabilities related to lack of proper monitoring, improper dependency management, and poor secret management. Let’s dive right in!
No Real-Time Monitoring
A timely, real-time response is paramount in the event of an attack. Every second counts. The longer an attacker remains unnoticed, the greater her advantage and the consequences of her attack.
What you can do…
Real-time, accurate, and fine-grained monitoring and logging is essential. To achieve this, however, it is necessary to know what to monitor and log. The SANS Institute specifies the categories in which appropriate logging should be implemented:
- Authentication and Authorization: Log API access keys associated with logins (successful or not). Similarly, log any attempt to invoke any function with improper permissions
- Change: Log potentially dangerous changes such as the deployment of new functions (successful or not), permission changes on functions, cloud storage services, and trigger definitions
- Network Activity: Log outbound connections started by your functions
- Resource Access: Log every function execution and any access to external accounts that are not related to the primary account that owns the function
- Malware Activity: Log all activity initiated by third-party dependencies and any changes implemented to and by them
- Critical Errors and Failures: Log all crashes, timeouts, retries, and all execution limits that have been reached
Cloud providers already offer useful, though limited, tools, such as AWS X-Ray and Azure Monitor, to monitor and log activity in most of the categories above. Other third-party tools, such as Dashbird, build on top of the ones offered by cloud providers to give the user a more pleasant and complete experience.
Adding X-Ray tracing to a project is trivial with the Serverless Framework. See highlighted lines below:
15provider:
16 name: aws
17 runtime: go1.x
18 memorySize: 128
19 versionFunctions: false
20 stage: dev
21 region: us-east-1
22 tracing:
23 apiGateway: true
24 lambda: true
25 environment:
26 tableName: ${self:custom.tableName}There are a few other metrics that are very particular to serverless functions such as invocations count, number of crashes or timeouts, number of cold starts, memory utilization, and duration of execution.
When monitored correctly and promptly, they can offer valuable feedback that may drive decisions related to resource utilization, financial costs, third-party libraries (see next section), and implementation details and optimization.
There are situations in which monitoring and logging need to be implemented from scratch instead of relying on external tools. This brings additional concerns regarding the data collected by monitoring and logging tools. These concerns include but are not limited to:
- Collection: Is the API collecting data? Is collecting this data relevant, necessary, and safe?
- Security: Is the data being collected and stored securely?
- Access: Is the data accessed only by authorized consumers and in a secure way? Is this data providing a decent ROI?
- Segmentation: Is the data being overexposed? Are the consumers of this data correctly identified and granted the right access?
The advantage of using managed monitoring tools, such as CloudWatch, is that many of these issues are addressed by the Cloud provider.
Dangerous Third-party Dependencies
The usage of third-party dependencies should be minimized as much as possible. Utmost care needs to be taken to only use dependencies from trusted sources that have not been compromised. Remember, your serverless function is as vulnerable as the most vulnerable of its dependencies.
An excellent source of information about common vulnerabilities is the MITRE Common Vulnerabilities and Exposure. Go ahead and check the vulnerabilities found for your favorite language.
What you can do…
The first step is to keep a detailed inventory of all the dependencies used by your serverless functions. This also includes the whole dependency hierarchy brought by these dependencies. Then implement a process to constantly check for any vulnerabilities found for these dependencies and any new ones that are added. This in itself is a titanic task. Therefore, a tool such as Snyk comes handy.
This brings us to the most important point of this section. Make use of monitoring software to check for known vulnerable dependencies, especially when adding new dependencies or upgrading their versions.
Vulnerability scanning should be done as part of your ongoing CI/CD process. So that when a vulnerability is found in a dependency, the build/deploy process is interrupted until the application owner to upgrades/patches the affected dependency. Remember, since server dependencies and libraries are updated quickly by the cloud provider, attackers will focus their attention on application packages when looking for vulnerabilities to exploit.
Once again, tools like Snyk provide a world-class service when guarding against vulnerable dependencies in serverless projects. Getting this service up and running in your project is extremely simple. More details can be found here.
In summary, automatic monitoring should not be limited to running serverless functions. It starts the moment the first line of code is written. Automatic monitoring allows for a timely response in the event of finding compromised or deprecated dependencies and the removal of unnecessary dependencies, especially when such dependencies are no longer required by your serverless functions.
Improper Handling Of Sensitive Information
How well do your serverless functions keep secrets, such as API Keys, DB credentials, encryption keys, and configuration settings? How are these secrets handled and stored?
By default, serverless functions are loosely coupled and deployed independently from each other. Therefore, they do not have a centralized way to share sensitive, common information, such as configuration, between them.
A common way to do this is through environment variables. However, if not properly handled, they can easily leak the information they are supposed to protect.
Specifically, even though the information contained in environment variables is automatically encrypted and decrypted after deployment and during function execution, this information is not encrypted and therefore completely accessible before and during deployment.
What you can do…
If secrets (sensitive information) are going to be kept in environment variables, they should be properly encrypted before deployment (client-side). Decryption should only occur during execution and by using a proper key management system such as AWS KMS or Azure Key Vault. A great tutorial on Lambdas with KMS can be found here.
Client-side variable encryption is easy to do in AWS using the Lambda console and encryption helpers that leverage AWS KMS. A more automated and repeatable way of doing can be implemented using the Serverless Framework.
First things first, create a Customer Master Key. You can do this through the AWS Console, the AWS client, or tools like Terraform. The Terraform version is shown below:
1resource "aws_kms_key" "a" {
2 description = "Sample KMS"
3 deletion_window_in_days = 7
4}It cannot get simpler than that. We provide a brief description and the duration in days after which the key is deleted after the destruction of the resource.
With the key in hand, we can now proceed to encrypt the variable or variables that need to be encrypted so they can be safely passed to the Serverless framework for deployment. For this, we are going to need either the ID or ARN of our newly created key. With that in hand, we can run the following KMS encryption command:
$ aws kms encrypt --key-id 'arn:aws:kms:...' --plaintext 'theSecret'The output of this command contains the CiphertextBlob field whose value is a Base64-encoded string representing the encrypted version of the value referenced by the plainText parameter. This string will be passed as an environment variable in the Serverless framework deployment script, as shown below.
1 service: aws-lambda-lab
2
3 frameworkVersion: ">=1.28.0 <2.0.0"
4
5 custom:
6 stage: ${opt:stage, self:provider.stage}
7
8 provider:
9 name: aws
10 runtime: go1.x
11 memorySize: 128
12 versionFunctions: false
13 stage: dev
14 region: us-east-1
15 tracing:
16 apiGateway: true
17 lambda: true
18 environment:
19 password: AQICAHikasXx5gVVjFkLuIk8ntupVE91r7p...
20
21 package:
22 exclude:
23 - ./**
24 include:
25 - ./bin/**
26
27 functions:
28 guessSecret:
29 handler: bin/guesssecret
30 events:
31 - http:
32 path: guesssecret
33 method: post
34 cors: true
35 authorizer: aws_iam
36 awsKmsKeyArn: arn:aws:kms:us-east-1...
37
38 resources:
39 - ${file(resources/api-gateway-errors.yml)}
40 - ${file(resources/cognito-user-pool.yml)}
41 - ${file(resources/cognito-identity-pool.yml)}The highlighted lines in the snippet above show the encrypted environment variable that is passed to the Lambda function and its corresponding KMS key. The Lambda function requires the reference to the KMS key that was used to encrypt the environment variable so it can decrypt it appropriately.
Finally, once the key and deployment script are in place, all that is left is the implementation of the logic to decrypt the secret environment variable. The snippet below shows how to do this in Go. For more details, refer to the corresponding file in the repository.
27kmsSvc := kms.New(session.Must(session.NewSession()))
28// Get encrypted password from environment
29encryptedPassword, err := b64.URLEncoding.DecodeString(os.Getenv("password"))
30if err != nil {
31 return Response{StatusCode: 400}, err
32}
33input := &kms.DecryptInput{
34 CiphertextBlob: encryptedPassword,
35}
36
37// Decode encrypted password
38result, err := kmsSvc.Decrypt(input)
39if err != nil {
40 return Response{StatusCode: 500}, err
41}
42
43// Do something with plain text password value
44isSecretValueCorrect := apiEvent.ControlValue == string(result.Plaintext)
Wrapping Up…
This post picks up where the previous post left off. That is, we continue to check the list of serverless functions security best practices.
We analyzed the importance of proper monitoring to detect potential attacks on time. We looked at the problems of using dangerous third-party dependencies and the available tools that can help us detect them. Finally, we checked the benefits of properly handling sensitive information (secrets) in serverless functions by using cloud-native tools such as AWS KMS.
Companion repository for this post can be found at https://github.com/jpcedenog/blog-harden-serverless-api-next-level
Thanks for reading!
Image by PublicDomainPictures from Pixabay