Issue description
We are facing intermittent issues resolving dynamic variables—especially SSM/SecretsManager-based secrets and stage/environment-dependent values—when using Serverless Framework versions beyond v4.4.3.
These failures occur both during serverless deploy -s dev and serverless offline start -s dev. About 70% of the time, variable resolution fails, while the remaining attempts succeed without any changes to configuration or credentials.
We're now suspecting the issue is not limited to SSM resolution alone, but could be linked to dynamic variable interpolation (e.g., ${self:custom.ssm.${self:provider.stage}.openai-api-key}) or stage/environment usage across files.
Error:
✖ Error: Cannot resolve '${self:custom.ssm.dev.openai-api-key}' variable at 'provider.environment.OPENAI_API_KEY'. No value is available for this variable, and no default value was provided. Please check your variable definitions or provide a default value.
at _ResolverManager.resolve (file:///home/user/.serverless/releases/4.4.4/package/dist/sf-core.js:655:55913)
at async #handlePlaceholderNode (file:///home/user/.serverless/releases/4.4.4/package/dist/sf-core.js:655:54343)
at async file:///home/user/.serverless/releases/4.4.4/package/dist/sf-core.js:655:52335
at async processNodeAndHandleCompletion (file:///home/user/.serverless/releases/4.4.4/package/dist/sf-core.js:655:43086)
Environment:
- Node.js: v22.15.0
- Serverless Framework: Affected on 4.4.4, 4.14.3, 4.17.0, 4.17.1 (works on 4.4.2)
- AWS CLI v2, using AWS SSO login via aws-sso-credential-getter
Project Configuration Overview:
// serverless.yml:
custom: ${file(serverless_custom.yml)}
provider:
name: aws
profile: ${env:my-aws-profile}
runtime: nodejs20.x
stage: ${opt:stage, 'dev'}
region: ${opt:region, 'us-east-1'}
environment: ${file(serverless_environment.yml)}
// serverless_environment.yml:
OPENAI_API_KEY: ${self:custom.ssm.${self:provider.stage}.openai-api-key}
MOMENTO_API_KEY: ${self:custom.ssm.${self:provider.stage}.momento-api-key}
S3_UPLOAD_BUCKET: spicychat-assets-${self:provider.stage}
FIREHOSE_STREAM_NAME: ${self:custom.firehoseStreamName.${self:provider.stage}}
// serverless_custom.yml:
ssm:
prod: ${ssm:/aws/reference/secretsmanager/spicychat/${sls:stage}}
dev: ${ssm:/aws/reference/secretsmanager/spicychat/${sls:stage}}
uat: ${ssm:/aws/reference/secretsmanager/spicychat/${sls:stage}}
Steps to Reproduce:
export AWS_PROFILE=<my-aws-profile>
aws sso login --profile <my-aws-profile>
ssocred <my-aws-profile>
serverless deploy -s dev
OR
serverless offline start -s dev
- Observe that sometimes all environment variables resolve correctly, but most times critical ones like OPENAI_API_KEY, MOMENTO_API_KEY, etc., are resolved as null.
What we've tried
- Verified all secrets exist in AWS Secrets Manager.
- Cleared
.serverless cache directory.
- Confirmed valid SSO session and that AWS profile environment variables are set.
- Reverted to v4.4.2 — the last known stable version where secrets resolve consistently.
Expected Behavior:
Serverless should consistently resolve SSM secrets when valid AWS credentials are provided and referenced correctly in the configuration.
Actual Behavior:
SSM secrets intermittently fail to resolve, leading to deployment and local development failures. Variables resolve to null, and placeholders like ${self:custom.ssm.dev.openai-api-key} trigger fatal errors due to unresolved values.
Debug Logs:
s:core: { serverless_framework: '4.4.4' }
s:main: Initializing
s:core:router: {
command: [ 'offline', 'start' ],
options: { s: 'dev', debug: '*' },
versions: { serverless_framework: '4.4.4' },
customConfigFilePath: null,
composeParams: undefined,
composeResolverProviders: undefined,
isWithinCompose: false,
composeOrgName: null,
composeServiceName: null
}
.........
.........
.........
s:core:resolver:manager: adding resolver provider opt with config { type: 'opt' }
s:core:resolver:manager: provider 'opt', resolver 'options' resolved 'region' to 'null' - skipping to next fallback.
s:core:resolver:manager: resolved ${opt:region, 'us-east-1'} to us-east-1
s:core:auth:get-authenticated-data: license key provided manually
s:core:resolver:manager: adding resolver provider env with config { type: 'env' }
s:core:resolver:manager: resolved ${env:my-aws-profile} to
s:core:resolver:manager: adding resolver provider file with config { type: 'file' }
s:core:resolver:manager: adding resolver provider self with config { type: 'self' }
s:core:resolver:manager: resolved ${opt:stage, 'dev'} to dev
s:core:resolver:manager: resolved ${opt:stage, 'dev'} to dev
s:core:resolver:manager: resolved ${self:service} to spicychat-backend
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${file(serverless_custom.yml)} to [object Object]
s:core:resolver:manager: adding resolver for aws credential resolver ssm
s:core:resolver:manager: adding resolver provider sls with config { type: 'sls' }
s:core:resolver:manager: resolved ${self:custom.cors.allowCredentials.dev, ''} to false
s:core:resolver:manager: resolved ${self:custom.cors.allowedOrigins.dev} to *
s:core:resolver:manager: resolved ${sls:stage} to dev
s:core:resolver:manager: resolved ${sls:stage} to dev
s:core:resolver:manager: resolved ${sls:stage} to dev
s:core:resolver:default-aws-credential-resolver: resolving AWS credentials using the AWS SDK credential provider chain with profile:
s:core:resolver:manager: resolved ${file(src/character/translation/stateMachine.yml)} to [object Object]
s:core:resolver:manager: adding resolver provider aws with config { type: 'aws' }
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${file(serverless_environment.yml)} to [object Object]
s:core:resolver:manager: adding resolver for aws credential resolver cf-spicychat-backend-${self:provider.stage}.ApiId
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:service} to spicychat-backend
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.region} to us-east-1
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.region} to us-east-1
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:service} to spicychat-backend
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: resolved ${self:provider.region} to us-east-1
s:core:resolver:manager: resolved ${self:provider.stage} to dev
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.openai-api-key' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.momento-api-key' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.typesense-api-key' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.typesense-public-api-key' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.typesense-not-public-api-key' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.typesense-activity-api-key' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.jwt-signing-key' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.myscale-password' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.CLICKHOUSE_PASSWORD' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.recombee-private-token' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.tapfiliate-key' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.stickyio-credentials' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.contentful-management-token' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.contentful-api-key' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.axiom-token' to 'null' - skipping to next fallback.
s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.SIGHT_ENGINE_API_SECRET' to 'null' - skipping to next fallback.
s:core:resolver:manager: resolved ${self:provider.environment.WAITINQUEUE_SQS} to spicychat-waitingqueue-dev
s:core:resolver:manager: resolved ${self:custom.firehoseStreamName.dev} to spicychat-stream-v2-dev
s:core:resolver:manager: resolved ${self:custom.axiomDataset.spicychat-backend} to nextday-backend
s:core:meta: Skipping analysis event because license key is used
s:core:meta: saving meta record to global .serverless directory.
s:core:resolver:manager: resolved ${file(./serverless-stacks/spicychat-backend/sqs.yml)} to [object Object]
s:core:resolver:manager: resolved ${file(./serverless-stacks/spicychat-backend/lambda-functions.yml)} to ${file(src/chat/queue/config.yml)},${file(src/chat/queue/routes.yml)},${file(src/chat/message/routes.yml)},${file(src/chat/conversation/routes.yml)},${file(src/chat/conversation/config.yml)},${file(src/chat/domain/routes.yml)},${file(src/chat/voice/domain/routes.yml)},${file(src/m2m/routes.yml)},${file(src/order/routes.yml)},${file(src/user/persona/routes.yml)},${file(src/user/domain/routes.yml)},${file(src/user/domain/config.yml)},${file(src/user/setting/routes.yml)},${file(src/subscription/config.yml)},${file(src/subscription/routes.yml)},${file(src/shared/system/config.yml)},${file(src/shared/crm/config.yml)},${file(src/shared/content/routes.yml)},${file(src/shared/image/routes.yml)},${file(src/shared/event/routes.yml)},${file(src/shared/growthbook/routes.yaml)},${file(src/character/domain/trigger/config.yml)},${file(src/character/recommendation/config.yml)},${file(src/character/domain/routes.yml)},${file(src/character/translation/routes.yml)}
s:core:resolver:manager: resolved ${file(./serverless-stacks/spicychat-backend/dynamodb-tables.yml)} to [object Object]
s:core:resolver:manager: resolved ${aws:region} to us-east-1
s:cli-error-handler: Error: Cannot resolve '${self:custom.ssm.dev.openai-api-key}' variable at 'provider.environment.OPENAI_API_KEY'. No value is available for this variable, and no default value was provided. Please check your variable definitions or provide a default value.
at _ResolverManager.resolve (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:667:11)
at async _ResolverManager.#handlePlaceholderNode (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:584:9)
at async (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:477:11)
at async processNodeAndHandleCompletion (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/graph.js:20:5)
✖ Error: Cannot resolve '${self:custom.ssm.dev.openai-api-key}' variable at 'provider.environment.OPENAI_API_KEY'. No value is available for this variable, and no default value was provided. Please check your variable definitions or provide a default value.
at _ResolverManager.resolve (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:667:11)
at async _ResolverManager.#handlePlaceholderNode (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:584:9)
at async (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:477:11)
at async processNodeAndHandleCompletion (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/graph.js:20:5)