Allow storing test metrics in local filesystem of LocalStack container #13188

nik-localstack · 2025-09-24T13:55:07Z

Motivation

We want to collect test metrics when the system running pytest is not the same running LocalStack

Changes

Introduces the ability to store test metrics directly within the LocalStack container's filesystem instead of only in the system running pytest. This provides more flexibility for metric collection in different deployment scenarios (like when running in k8s)
Adds ENV_INTERNAL_TEST_STORE_METRICS_IN_LOCALSTACK environment variable to enable local filesystem storage
Adds ENV_INTERNAL_TEST_STORE_METRICS_PATH environment variable to specify custom storage path (defaults to /tmp/localstack-metrics)

github-actions · 2025-09-24T14:02:34Z

S3 Image Test Results (AMD64 / ARM64)

2 files ±0 2 suites ±0 8m 12s ⏱️ +17s
532 tests ±0 480 ✅ ±0 52 💤 ±0 0 ❌ ±0
1 064 runs ±0 960 ✅ ±0 104 💤 ±0 0 ❌ ±0

Results for commit ecdd453. ± Comparison against base commit a4e351d.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-24T14:15:16Z

Test Results - Preflight, Unit

22 291 tests ±0 20 548 ✅ ±0 15m 21s ⏱️ -56s
1 suites ±0 1 743 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit ecdd453. ± Comparison against base commit a4e351d.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-24T14:24:46Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 23s ⏱️ +2s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit ecdd453. ± Comparison against base commit a4e351d.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-24T15:08:11Z

Test Results (amd64) - Integration, Bootstrap

5 files 5 suites 2h 38m 49s ⏱️
5 169 tests 4 673 ✅ 496 💤 0 ❌
5 175 runs 4 673 ✅ 502 💤 0 ❌

Results for commit ecdd453.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-24T15:13:16Z

LocalStack Community integration with Pro

2 files ±0 2 suites ±0 2h 0m 30s ⏱️ +30s
4 795 tests ±0 4 459 ✅ ±0 336 💤 ±0 0 ❌ ±0
4 797 runs ±0 4 459 ✅ ±0 338 💤 ±0 0 ❌ ±0

Results for commit ecdd453. ± Comparison against base commit a4e351d.

♻️ This comment has been updated with latest results.

cloutierMat

Thanks for tackling this! 🚀 Overall the changes look good and I like the use of constants here instead of hardcoded strings 👍

I am not sure about the file saving strategy, we now lose the de-duplicate and we are doing file operations on every request. I would like have a bit more insight over the impact of those changes before merging.

cloutierMat · 2025-10-06T15:56:38Z

localstack-core/localstack/aws/handlers/metric_handler.py

        # refrain from adding duplicates
        if metric not in MetricHandler.metric_data:
-            MetricHandler.metric_data.append(metric)
+            self.append_metric(metric)


question: Since when storing the file locally, we don't append to MetricHandler.metric_data this won't prevent duplicate. Do you know if the risk of duplicate entry is high? The pr that introduces it doesn't give much more information.

I don't understand the need of adding duplicates. In the localstack-docs script https://github.com/localstack/localstack-docs/blob/main/scripts/create_data_coverage.py that uses these files, duplication is not affecting the results.

I am not sure either, it is also only called from a single response handler, not too sure how there could even be a duplicate in the first place 🤷‍♂️

cloutierMat · 2025-10-06T16:24:13Z

localstack-core/localstack/aws/handlers/metric_handler.py

+            with open(self.local_filename, "a") as fd:
+                writer = csv.writer(fd)
+                writer.writerow(metric)


question: My understanding is that every request going through our gateway will create an entry and open/close the csv file. Have you tested the cost implication of this? Do you have any full run example showing the additional cost is minimal?

The tests may take ~3 minutes more than before so the impact is not that big.
We can reiterate and improve later if it starts becoming an issue, wdyt ?

Sounds reasonable enough 😄

nik-localstack

Thanks for the review @cloutierMat
I agree with your concerns, and I tried to think of a better approach but given that this flow is a bit outdated (and most of the data that are created from these report files are not used anymore) I don't think it makes sense to invest more time now. Ideally we can re-iterate if this becomes an issue, or if/when we clean up this process a bit.

nik-localstack · 2025-10-07T08:05:43Z

localstack-core/localstack/aws/handlers/metric_handler.py

+            with open(self.local_filename, "a") as fd:
+                writer = csv.writer(fd)
+                writer.writerow(metric)


The tests may take ~3 minutes more than before so the impact is not that big.
We can reiterate and improve later if it starts becoming an issue, wdyt ?

nik-localstack · 2025-10-07T08:07:39Z

localstack-core/localstack/aws/handlers/metric_handler.py

        # refrain from adding duplicates
        if metric not in MetricHandler.metric_data:
-            MetricHandler.metric_data.append(metric)
+            self.append_metric(metric)


I don't understand the need of adding duplicates. In the localstack-docs script https://github.com/localstack/localstack-docs/blob/main/scripts/create_data_coverage.py that uses these files, duplication is not affecting the results.

cloutierMat

Thanks for answering my concerns @nik-localstack!

🚀 LGTM

cloutierMat · 2025-10-07T18:19:51Z

localstack-core/localstack/aws/handlers/metric_handler.py

        # refrain from adding duplicates
        if metric not in MetricHandler.metric_data:
-            MetricHandler.metric_data.append(metric)
+            self.append_metric(metric)


I am not sure either, it is also only called from a single response handler, not too sure how there could even be a duplicate in the first place 🤷‍♂️

cloutierMat · 2025-10-07T18:20:20Z

localstack-core/localstack/aws/handlers/metric_handler.py

+            with open(self.local_filename, "a") as fd:
+                writer = csv.writer(fd)
+                writer.writerow(metric)


Sounds reasonable enough 😄

#13188)

nik-localstack self-assigned this Sep 24, 2025

nik-localstack added semver: patch Non-breaking changes which can be included in patch releases docs: skip Pull request does not require documentation changes labels Sep 24, 2025

alexrashed added the notes: skip Pull request does not have to be mentioned in the release notes label Sep 24, 2025

nik-localstack force-pushed the k8s/tests-api-coverage branch from 316165c to d3b685b Compare September 25, 2025 07:57

nik-localstack added this to the 4.10 milestone Sep 26, 2025

nik-localstack force-pushed the k8s/tests-api-coverage branch 3 times, most recently from 578366d to 1fda6e1 Compare October 2, 2025 12:54

Allow storing test metrics in local filesystem of LocalStack container

ecdd453

nik-localstack force-pushed the k8s/tests-api-coverage branch from 1fda6e1 to ecdd453 Compare October 6, 2025 09:11

nik-localstack marked this pull request as ready for review October 6, 2025 14:51

nik-localstack requested a review from thrau as a code owner October 6, 2025 14:51

nik-localstack requested a review from a team October 6, 2025 14:51

cloutierMat reviewed Oct 6, 2025

View reviewed changes

nik-localstack commented Oct 7, 2025

View reviewed changes

cloutierMat approved these changes Oct 7, 2025

View reviewed changes

nik-localstack merged commit 343d436 into main Oct 13, 2025
48 checks passed

nik-localstack deleted the k8s/tests-api-coverage branch October 13, 2025 06:20

baermat pushed a commit that referenced this pull request Oct 13, 2025

Allow storing test metrics in local filesystem of LocalStack container (

b3b7ab4

#13188)

Uh oh!

Allow storing test metrics in local filesystem of LocalStack container #13188

Allow storing test metrics in local filesystem of LocalStack container #13188

Uh oh!

Conversation

nik-localstack commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

github-actions bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

S3 Image Test Results (AMD64 / ARM64)

Uh oh!

github-actions bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Preflight, Unit

Uh oh!

github-actions bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

github-actions bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

cloutierMat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nik-localstack left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloutierMat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nik-localstack commented Sep 24, 2025 •

edited

Loading

github-actions bot commented Sep 24, 2025 •

edited

Loading

github-actions bot commented Sep 24, 2025 •

edited

Loading

github-actions bot commented Sep 24, 2025 •

edited

Loading

github-actions bot commented Sep 24, 2025 •

edited

Loading

github-actions bot commented Sep 24, 2025 •

edited

Loading