Opensearch: Ignore SSL validation for ES plugin test #13234

nik-localstack · 2025-10-07T11:00:56Z

Motivation

Fix failing tests from https://github.com/localstack/localstack-pro/pull/5371

The test currently fails when it runs against LocalStack running in k8s.

Changes

Skip SSL validation when reaching the Elasticsearch plugins endpoint in test.

Context

It seems that something like *.{region}.es.localhost.localstack.cloud should be added in our certificate creation logic but there is already a comment in the test that SSL validation is skipped.

github-actions · 2025-10-07T11:20:39Z

Test Results - Preflight, Unit

22 292 tests ±0 20 549 ✅ ±0 15m 29s ⏱️ -5s
1 suites ±0 1 743 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 4f7f972. ± Comparison against base commit 928f470.

github-actions · 2025-10-07T11:21:24Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 9m 33s ⏱️ - 1h 49m 30s
48 tests - 4 747 33 ✅ - 4 426 15 💤 - 321 0 ❌ ±0
50 runs - 4 747 33 ✅ - 4 426 17 💤 - 321 0 ❌ ±0

Results for commit 4f7f972. ± Comparison against base commit 928f470.

This pull request removes 4747 tests.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

github-actions · 2025-10-07T11:32:55Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 21s ⏱️ -1s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 4f7f972. ± Comparison against base commit 928f470.

github-actions · 2025-10-07T11:38:40Z

Test Results (amd64) - Integration, Bootstrap

5 files ± 0 5 suites ±0 19m 7s ⏱️ - 2h 19m 26s
72 tests - 5 097 57 ✅ - 4 616 15 💤 - 481 0 ❌ ±0
78 runs - 5 097 57 ✅ - 4 616 21 💤 - 481 0 ❌ ±0

Results for commit 4f7f972. ± Comparison against base commit 928f470.

This pull request removes 5097 tests.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

alexrashed · 2025-10-07T12:30:41Z

tests/aws/services/opensearch/test_opensearch.py

        else:
            # TODO fix ssl validation error when using the signed request for the elastic search domain
-            plugins_response = requests.get(plugins_url, headers={"Accept": "application/json"})
+            plugins_response = requests.get(


I am a bit confused by this change, why does it work in the current test pipeline just here in LocalStack Community? Isn't this what the if/else here is for? Is this change really necessary?

I'm also a bit confused and cannot understand why it passes in the current pipeline.
In my local env the test fails when running it against localstack-pro container. When I add the verify=False it works.

The long term solution is to fix the certificate domains with https://github.com/localstack/localstack-certificate-infrastructure/pull/4 but this is a bit out of scope of the current project (collecting coverage of operations when running LocalStack on k8s)

In the current pipeline, we don't face the same problem because of this

localstack/localstack-core/localstack/testing/pytest/in_memory_localstack.py

Line 76 in cd7b481

safe_requests.verify_ssl = False

that sets the ssl verification off
So this change enforces the same behavior regardless if the tests run with in memory localstack or against an "external" localstack container / pod

thought (non-blocking): Super interesting and great find!
Do you think it would make sense / would be possible to remove this line though? I am asking because I think it is problematic to change a test here because it is run in a pipeline which is not used as a quality gate here, i.e. there's nothing preventing us from breaking this again, or for a different test in the future.
I would love to make sure that we a guarded against a regression like this in the future.
I think this might be a remnant from a time where we did not serve both HTTP and HTTPS on 4566, so maybe you are actually solving the "same problem" here, i.e. I think if your external pipeline runs against the tests here, the verify_ssl could be set to True without any implications.
In fact, changing the variable here and making sure the test pipeline is still green here in this repo could maybe even speed up the development of your external pipeline. What do you think?

Happy to discuss this specifically, but I also don't want to block this PR in case you want to look at this in a future iteration.

I will merge this now to unblock the remaining parts.

I think we will be able to remove this line once/when we update our certificate.

I will get back to this next week.

nik-localstack · 2025-10-07T12:56:53Z

Adding @cloutierMat (as author of the test) as a reviewer in case you have some more context

alexrashed

Thanks a lot for digging into why this is currently working in the test pipeline! 🕵🏽
As I mentioned in the comment, I think it might be worth changing the config of the tests and combining this change with some potential other fixes among the codebase, but I will directly approve this PR as it is right now because it's totally up to you if you want to tackle this in other PRs / in another iteration. Let me know if I can help anywhere! :)

alexrashed · 2025-10-07T13:52:13Z

tests/aws/services/opensearch/test_opensearch.py

        else:
            # TODO fix ssl validation error when using the signed request for the elastic search domain
-            plugins_response = requests.get(plugins_url, headers={"Accept": "application/json"})
+            plugins_response = requests.get(


thought (non-blocking): Super interesting and great find!
Do you think it would make sense / would be possible to remove this line though? I am asking because I think it is problematic to change a test here because it is run in a pipeline which is not used as a quality gate here, i.e. there's nothing preventing us from breaking this again, or for a different test in the future.
I would love to make sure that we a guarded against a regression like this in the future.
I think this might be a remnant from a time where we did not serve both HTTP and HTTPS on 4566, so maybe you are actually solving the "same problem" here, i.e. I think if your external pipeline runs against the tests here, the verify_ssl could be set to True without any implications.
In fact, changing the variable here and making sure the test pipeline is still green here in this repo could maybe even speed up the development of your external pipeline. What do you think?

Happy to discuss this specifically, but I also don't want to block this PR in case you want to look at this in a future iteration.

alexrashed · 2025-10-07T15:08:03Z

I think we will be able to remove this line once/when we update our certificate.

I think there is a misunderstanding. I think we might not update our certificate for this (because we can only have a limited amount of entries in there, and elasticsearch might not be worth it), and I think we can remove the line you identified as fixing this in the pipeline, but replacing the individual requests with their "unverified" calls.
This is effectively what you are doing right now, and once all these tests / calls run in your pipeline, this line will be obsolete:

localstack/localstack-core/localstack/testing/pytest/in_memory_localstack.py

Line 76 in cd7b481

safe_requests.verify_ssl = False

So what I am saying here is: Remove this line now, see which tests are breaking here, fix these calls with verify=False, and you will have:

Your external pipeline being fixed.
This pipeline fail in case someone would introduce a call which would not run in your pipeline.

Really happy to discuss this directly if you want :)

Opensearch: Ignore SSL validation for ES plugin test

4f7f972

nik-localstack added semver: patch Non-breaking changes which can be included in patch releases docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes labels Oct 7, 2025

nik-localstack added the aws:opensearch Amazon OpenSearch Service label Oct 7, 2025

nik-localstack marked this pull request as ready for review October 7, 2025 11:26

nik-localstack requested review from alexrashed and silv-io as code owners October 7, 2025 11:26

alexrashed reviewed Oct 7, 2025

View reviewed changes

nik-localstack requested a review from cloutierMat October 7, 2025 12:56

alexrashed approved these changes Oct 7, 2025

View reviewed changes

nik-localstack merged commit 19c5bf0 into main Oct 7, 2025
59 of 61 checks passed

nik-localstack deleted the k8s/run-tests-for-community-2 branch October 7, 2025 14:12

baermat pushed a commit that referenced this pull request Oct 13, 2025

Opensearch: Ignore SSL validation for ES plugin test (#13234)

aa8c0f8

nik-localstack mentioned this pull request Oct 16, 2025

Remove verify_ssl = False from in memory localstack test configuration #13275

Merged

Uh oh!

Opensearch: Ignore SSL validation for ES plugin test #13234

Opensearch: Ignore SSL validation for ES plugin test #13234

Uh oh!

Conversation

nik-localstack commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Context

Uh oh!

github-actions bot commented Oct 7, 2025

Test Results - Preflight, Unit

Uh oh!

github-actions bot commented Oct 7, 2025

LocalStack Community integration with Pro

Uh oh!

github-actions bot commented Oct 7, 2025

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Oct 7, 2025

Test Results (amd64) - Integration, Bootstrap

Uh oh!

alexrashed Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

nik-localstack Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

nik-localstack Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

alexrashed Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

nik-localstack Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

nik-localstack commented Oct 7, 2025

Uh oh!

alexrashed left a comment

Choose a reason for hiding this comment

Uh oh!

alexrashed Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexrashed commented Oct 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nik-localstack commented Oct 7, 2025 •

edited

Loading