latest newsVertex AIbucket squattingML supply chainpickle deserializationcloud SDKsupply chain attackGoogle Cloud

The Next Five Years of ML Supply Chain Attacks Start With Predictable Cloud SDKs

Geert WarmenbolGeert WarmenbolJune 17, 2026
Share:
The Next Five Years of ML Supply Chain Attacks Start With Predictable Cloud SDKs

In March 2026, a researcher at Unit 42 noticed something odd. The Vertex AI SDK for Python, when told to upload a model without a custom staging bucket, generated a name like project-vertex-staging-us-central1. That name was predictable and globally unique across all Google Cloud projects. Anyone who created that bucket first owned the staging path. The researcher built a proof of concept: spin up a Cloud Function triggered by the object upload event, swap the model file with a malicious pickle payload in under 1.4 seconds, and wait for Vertex AI to load it. The 2.5 second gap between upload and load was enough. Google patched it. But the pattern did not die. It simply moved to the next SDK, the next cloud provider, the next default. The incident revealed something deeper than a bug: cloud SDKs are designed for convenience, not adversarial trust models. Every default bucket name, every auto-generated resource identifier, is an invitation for an attacker who can predict the namespace.

CASE RECONSTRUCTION

A mid-size fintech company runs a fraud detection model on Vertex AI. The ML team, under pressure to ship, uses the SDK defaults: no custom staging bucket, no custom service account. The project ID fintech-fraud-2026 is visible in their public documentation. An attacker creates fintech-fraud-2026-vertex-staging-us-central1 in their own project. They wait. The victim runs model.upload() after training. The SDK checks if the bucket exists: yes. It uploads the model artifacts. The attacker's Cloud Function triggers, downloads the artifact, appends a pickle payload that steals the compute environment's metadata credentials, and re-uploads. Vertex AI loads the model for serving. The pickle executes. The attacker now has the serving container's GCP metadata token. That token, in the default configuration, has access to BigQuery tables containing transaction records, model training data, and the GKE cluster. The attacker exfiltrates 50GB of labeled transaction data, enough to reverse-engineer the fraud model. The company discovers the breach only when the monthly cloud bill shows unexpected egress to an unknown IP. The incident timeline: 3 minutes from upload to full token theft. No alerts fired because the bucket was 'owned' by the attacker's project and the Cloud Function used a legitimate trigger. The ML team believed they followed best practices by using the SDK. They did not know the SDK's bucket choice was the vulnerability.

VULNOX FINDINGS

During five internal assessment cycles with Vulnox clients between February and May 2026, we observed the following patterns across cloud-native ML environments: 1) 6 out of 10 assessed environments used default staging bucket names for at least one production model deployment. 2) The average gap between upload and load in a real-world test was 1.9 seconds, not the 2.5 seconds reported by Unit 42, because Vertex AI pre-warms containers. That makes the race window even tighter. 3) In one assessment of an e-commerce recommendation system (team size: 8 ML engineers, one platform SRE), the default bucket had been in use for 14 months. The team explicitly said they 'trusted the SDK to handle infrastructure.' This trust extended to not monitoring bucket creation events in the cloud audit logs. 4) Counterintuitive: teams that used Infrastructure-as-Code (Terraform, Pulumi) were actually more likely to use default buckets because they templated the SDK calls without parameterizing the staging bucket argument. 5) Most surprising: three organizations had custom staging buckets, but those buckets were in a shared project controlled by a different team. The attack surface simply shifted from one project to another. The SDK still did not verify ownership of the bucket it was about to write to. The fix in v1.148.0 adds bucket ownership verification, but only for Python SDK. The Node.js and Java SDKs remain unpatched as of June 2026.

The vulnerability lives in the Python SDK's bucket selection logic. When a user calls model.upload() without specifying staging_bucket, the code constructs a name using f"{project}-vertex-staging-{region}". It then calls storage.Client().bucket(name) and checks bucket.exists(). The exists() call returns True if any bucket with that name exists in any project. The SDK does not verify that the bucket is in the same project or that the caller has write access. This design choice prioritizes flexibility (the bucket could be shared) but creates a trust gap. The attacker simply calls gsutil mb gs://victim-project-vertex-staging-us-central1 from their own project. Cloud Storage bucket creation is globally atomic: the first request wins. After the victim's model file lands, the attacker can replace it before Vertex AI's model loading routine reads it. The replacement payload, if pickle-serialized, executes arbitrary Python in the serving container. The exact code snippet that caused the issue: python def _get_default_staging_bucket(self, project, region): name = f"{project}-vertex-staging-{region}" bucket = self.storage_client.bucket(name) if bucket.exists(): return bucket bucket = self.storage_client.create_bucket(name, project=project) return bucket The bucket.exists() call checks for the bucket globally. Google's fix added a project parameter to the exists() call to scope it to the caller's project, and then added explicit project and caller identity verification before writing.

Prevention

Most advice tells you to update the SDK. That is necessary but not sufficient. The real fix requires changing how your team thinks about resource namespacing. Here are the steps ordered by impact.

RECOVERY PRIORITIES

If you discover that your model was uploaded to an attacker-controlled bucket, the order of your response determines whether you lose data, credentials, or both. First priority: Revoke the service account's token that was used for the upload. The attacker likely obtained the metadata server token. Force a token refresh by recreating the compute instance or pod. Do not wait to investigate. Every minute the token is valid, the attacker can escalate. Second priority: Audit the attacker's bucket via Cloud Storage logs. The bucket exists in their project. Ask Google Cloud support to identify the project ID. This is possible through support tickets. Do not assume you cannot identify the attacker. The bucket name includes your project ID. Third priority: Rotate all secrets that were accessible from the metadata server token. That includes service account keys for any Cloud Function, Cloud Run, or GKE workload that shared the same metadata endpoint. The token in Unit 42's test could access tenant-wide resources. Assume your token had similar scope. Fourth priority: Check for any model artifacts the attacker might have modified. Re-upload all production models from a known good source. Use hash verification. Fifth priority: Update the SDKs and add bucket ownership verification to all pipelines. This is the fix, but it is not the highest priority during active compromise. The highest priority is revoking the token. Get the order wrong and you could alert the attacker before you cut their access.

This vulnerability is not a bug. It is a design philosophy that prioritizes developer experience over security. Cloud SDKs are built for a world where the user is trusted and the resources are within one project. That world ended when multi-tenant bucket namespaces became a global attack surface. The industry needs to stop treating default resource names as a convenience. Every default name should be salted with a high-entropy string by default, not as an afterthought. The counterargument is that Google fixed it quickly. But the Node.js and Java SDKs remained unpatched for two months after the Python fix. That gap tells you the priority: Python is the ML language, so they fixed it first. The other SDKs are used by less security-critical workloads. That is a dangerous assumption. The Node.js SDK is used for backend services that call Vertex AI. Those services could be exploited in the same way. The industry should adopt a standard: any SDK that creates a cloud resource using a name derived from user-supplied data must verify ownership of that resource before writing to it. This should be part of the cloud provider's security baseline, not a per-product bounty.

FUTURE THREATS

The bucket squatting pattern will evolve into automated supply chain poisoning within two years. I predict that by Q3 2028, we will see a worm that scans public-facing documentation, GitHub repos, and error messages for project IDs, then automatically creates predictable staging buckets across multiple cloud providers (AWS S3 bucket names are also globally unique). The worm will wait for the first upload, inject a backdoor, and let the model be served for weeks before triggering. The enabling condition is the increasing use of cloud SDKs in CI/CD pipelines that run without human oversight. The attack does not need to be fast. It can be patient. A second scenario: the same predictable naming pattern exists in other cloud services. AWS SageMaker uses a default S3 bucket with a pattern sagemaker-region-account-id. That pattern is predictable but account-id is six digits plus a dash, so it is harder to guess. However, if an attacker obtains the account ID through a different leak (e.g., a public S3 bucket error message), they can pre-create the bucket in their own account. The enabling condition here is the cross-account bucket name collision. S3 bucket names are globally unique. If SageMaker does not verify ownership before writing, the same attack works. Google's fix to verify bucket ownership is a good start, but the industry-wide problem remains: SDKs assume the namespace is trusted. They should assume it is adversarial.

ACTION CHECKLIST

  1. (ML Engineer) Run grep -r 'staging_bucket' $(find . -name '*.py' -o -name '*.ipynb') in your ML codebase. Replace every unset staging bucket with an explicit bucket name that includes a random UUID. Outcome: immediate closure of the default bucket attack vector. 2. (Cloud Security Engineer) Enable storage.buckets.create audit logs and create a log sink with a filter for bucket names matching *-vertex-staging-* or *-sagemaker-*. Set up a Slack alert. Outcome: detect bucket squatting attempts in minutes. 3. (Platform Engineering Lead) Update the Python Vertex AI SDK to v1.148.0 or later across all runtime environments: notebooks, CI pipelines, Cloud Functions. Check the google-cloud-aiplatform version in your lock file. Outcome: limited fix, but necessary patch. 4. (ML Engineer) For any model that is loaded via pickle or joblib in production, add a pre-loading hash check. Store the expected SHA256 of the serialized model file in a separate secure location (e.g., Cloud KMS or a secrets manager). Verify the hash before calling pickle.load(). Outcome: blocks pickle-based code execution even if the file is replaced. 5. (Cloud Security Engineer) Review any custom buckets that were created before the patch. Check bucket IAM policies for unexpected bindings. Use gcloud storage buckets list --project=... and cross-reference with gsutil iam get gs://bucket. Outcome: identify any buckets that might have been pre-created by an attacker. 6. (CISO) Schedule a 30-minute threat model session focused on SDK defaults. Invite the ML team, the platform team, and a cloud security lead. Ask: what other SDKs do we use that create resources with predictable names? Outcome: surfaced counterpart vulnerabilities in other services before they are exploited.

Frequently Asked Questions

How can I check if my Vertex AI models were uploaded to a misconfigured bucket?

Look at the Cloud Storage audit logs for the `model.upload()` API call. Check the bucket name used in the metadata. If the bucket name matches the pattern `{project}-vertex-staging-{region}` and was created before your first upload, it could have been created by an attacker. You can also search your GCP audit logs for `google.cloud.aiplatform.v1.ModelService.UploadModel` and examine the `staging_bucket` parameter.

The fix only applies to Python SDK. Are Node.js or Java SDKs vulnerable?

Yes. As of June 2026, only the Python SDK (v1.148.0) has the ownership verification fix. The Node.js and Java Vertex AI SDKs still use the vulnerable bucket name pattern and do not verify ownership. If you use those SDKs in your ML pipeline, you must explicitly set `stagingBucket` or `staging_bucket` to a controlled bucket.

Can bucket squatting happen in other cloud services like AWS SageMaker?

Theoretically, yes. SageMaker creates a default S3 bucket with a name pattern `sagemaker-{region}-{account-id}`. Account IDs are numeric and sometimes leaked through error messages or public artifacts. If an attacker obtains the account ID and the bucket does not exist yet, they can create it in their own AWS account and intercept model artifacts. AWS requires bucket name uniqueness globally. SageMaker does not verify bucket ownership before writing. This is a known risk but has not been demonstrated publicly as of mid-2026.

Related Articles

Your AI Agent Has More Privilege Than the CEO: Why the Executor Is the Only Boundary That Matters

Your AI Agent Has More Privilege Than the CEO: Why the Executor Is the Only Boundary That Matters

Vulnox assessment data shows that in 9 of 11 enterprise agent deployments, the agent runtime had write access to its own identity provider. This article explains why tool lists, approval workflows, and guardrails fail, and how to enforce least privilege at the executor before a model rotation turns your agent into a domain admin.

The SSH Key That Nobody Killed Your Crypto Mining Response

The SSH Key That Nobody Killed Your Crypto Mining Response

This article walks through a real breach where a single unmanaged SSH key let a worm encrypt your network. You'll learn why Discord is the new C2, why root login is not your biggest problem, and what to do in 72 hours.

The Autonomous Agent Insider Crisis: What Your Detection Models Are Missing

The Autonomous Agent Insider Crisis: What Your Detection Models Are Missing

A Trellix survey of 600 CISOs shows 95% of companies report AI agent insider incidents. Vulnox assessment data reveals detection models built for humans miss machine-speed threats. This article covers the identity governance gap, credential management failures, and attack patterns that will define the next 24 months.

Ready to Secure Your Digital Assets?

Get a comprehensive vulnerability assessment for your website today.