Date: March 16th 2022
Security impact: Privilege Escalation, Data Leakage, Data Manipulation
This document is the vulnerability disclosure report once the vulnerability was discovered. For a description of the vulnerability, its impact and what should you do next, please refer to ZAPESCAPE: Organization-wide control over Code by Zapier
Step 1: Set Up an Endpoint to Exfiltrate Data
The endpoint will receive data, code and secrets exfiltrated from Zapier. For ease of use, we recommend using https://webhook.site/, which provides a dedicated endpoint viewable in the browser.
The website will generate a unique webhook for you to use. In this case, we got the unique url: https://webhook.site/0f340ac4-fd2b-4a4e-a0d4-0d973b2057a3.
Step 2: Create a Zap and add Code by Zapier
Create a new Zap. You can use a private Zap to avoid anyone including admins from detecting your malicious Zap. Choose any trigger. We used “Schedule by Zapier” and configured it to run hourly. Test your trigger.
When prompted to create an Action, choose “Code by Zapier” with Action Event “Run Python”.
The full Zap:
Step 3: Use a Malicious Payload to Take Control Over the Account’s Lambda instance
Insert the exfiltration URL to the code below. We have already done so with our unique URL from step 1 to the code.
Replace the “code” section with the payload below. The code is explained in detail later on.
In short, the code above replaces the Lambda runtime with our own malicious runtime version, and then invokes the Lambda Runtime API to process the current event and any future event, until Lambda’s cache gets recycled. Every time the code above is executed, the attacker gains full control over the entire account’s Lambda instance for several minutes. Of course, an attacker could set up a Zapier schedule to run this code every 10-15 minutes, thus gaining semi-persistency over the account’s Lambda instance.
Step 4: Zapier will Send Data, Code and Secrets to your Exfiltration URL
While our malicious runtime is handling events instead of AWS’s native runtime, we can do whatever we want. To demonstrate this capability, we introduce a malicious runtime which calls the method before handling any new event, and then continues to execute as a normal AWS’s runtime:
This produces a stream of data sent to our exfiltration URL by Zapier. The data contains the entire Lambda event, including
- Code by Zapier input
- The code used by the Code by Zapier action, including any secrets used, in particular those used by “Store by Zapier” or its Python client “StoreClient”.
On the Exfiltration Endpoint, we see:
Step 5: Run a Benign Zap
To see the attack in action, log in to Zapier with another user of that same Zapier account. Run any Zap that uses “Code by Zapier”, and watch the exfiltration URL for data leaked from Zapier. Once you trigger the Zap, Zapier would immediately send the Action input and code to the exfiltration URL.
Step 6: Automate to Maintain Access
Lambda environments are short-lived. Once the malicious code at step 4 is executed, the malicious runtime will process any event instead of the AWS runtime for as long as Lambda’s cache is not invalidated. Note that this could include several minutes where the original Lambda instance which processed the malicious payload has already terminated, but its cache is still valid. This means that each time the malicious code is executed, the attacker gains full access for several minutes. Our experiments show this to be 10-15 minutes.
To maintain access, an attacker could set up the malicious Zap to run every 10 or even 5 minutes, via a native schedule trigger or a webhook trigger to be triggered with another scheduler.
This allows the attacker to maintain full access for the vast majority of events processed by “Code by Zapier”.
This vulnerability enables gaining full control over the “Code by Zapier” runtime environment for the entire account. It is executable by ANY account user, no matter their role. The exploit could be introduced through a private Zap, which would ensure the attacker would not be detected, since admins would have no visibility into private Zaps other than knowing that they exist (even not the names).
Our research shows that “Code by Zapier” runs on an AWS Lambda function shared between the entire Zapier account with ARN “arn:aws:lambda:us-east-1:996097627176:function:prd-mngd-lmbd_paidcodeapipy37_ac
What can attackers do with this vulnerability? Really, anything they want within the scope of the Lambda function. The attacker is practically the function for the time of the attack. This includes:
- Exfiltrating Code by Zapier input and code (which most likely contains secrets, including those used by “Store by Zapier” or its Python client “StoraClient”).
- Steal AWS credentials – AWS passes an invoked role to Lambda via environment variables. In this case, Zapier’s lambda handler (stored at “/var/task/lambda_handler.py”) is actively deleting the following environment variables to avoid leakage: ’AWS_ACCESS_KEY_ID’, ‘AWS_SECURITY_TOKEN’, ‘AWS_SESSION_TOKEN’, ‘AWS_SECRET_ACCESS_KEY’. Of course, an attack with full control over the lambda runtime can replace Zapier’s lambda handler to disable deletion of those secrets, and steal them anyway.
- Manipulate the output of “Code by Zapier” to whatever the attacker would like.
- Use Zapier infrastructure for malicious purposes (e.g. DDOS).
- Steal secrets passed to “StoreClient” by replacing from store_api import StoreClient a malicious wrapper that sends the secret to an exfiltration endpoint. See Root Cause Analysis for context of this import command.
Note that since Zapier provides a separate Lambda function instance for each Zapier account, this vulnerability is scoped to intra-account privilege escalation.
Root Cause Analysis
Our research exposes that “Code by Zapier” runs input code with the following method:
Using the exec command without any limitation is extremely dangerous, and allows the attacker to, among other actions, spawn subprocesses, query the OS (as we did), communicate with the Lambda Runtime API (as done by the malicious code in step 2), and more.
To mitigate the issue, we identify two options. One solution would be for Zapier to provide a separate Lambda function instance for each user, thus avoiding one user gaining control over another user’s Lambda environment. Another solution would be to avoid using exec entirely, or more realistically to limit the scope of exec so it cannot spawn subprocesses or query the entire OS. We also recommend denying usage of the Python inspect module, which lets an attacker analyze the Python call stack for reconnaissance.
Additional Information – Recon
In order to demonstrate these capabilities, we first needed to use a few different commands for reconnaissance of the “Code by Zapier” environment. Using the Python inspect module, we explored to call stack to discover /var/task/lambda_handler.py, Zapier’s Lambda handler and /var/task/secret_store.py, Zapier’s wrapper around https://store.zapier.com. We were also able to peek into the caller stack, namely /var/runtime/bootstrap.py, to query the event and context objects, which are supposedly hidden from the exec command as seen in lambda_handler.py code above.
For more information about using RCE to gain full access over a Lambda function instance, see Gaining Persistency on Vulnerable Lambdas.