Skip to content

Conversation

@robinlehrmann
Copy link
Contributor

Hello :)

This will resolve #83

I thought it might be a good idea to parse the transports mapping in the messenger.yaml and map the transport with the eventSourceARN of a sqsEvent or the EventSubscriptionArn of a snsEvent with the configured transport.

With this solution the $transportName is not required anymore and I don't need to create one lambda per sqs queue for my use case :) Would be great if you can take a look at this @mnapoli

@robinlehrmann robinlehrmann force-pushed the transport-name-resolver branch from 3040cba to e59d589 Compare February 10, 2024 16:41
@tyx
Copy link
Contributor

tyx commented Feb 12, 2024

Hi !

I get your point but this is clearly a BC break according to your current implementation. We may continue to support old way and add possibility to introduce your concept that would fallback on old transportName if missing (and it would be nullable by defaut) ?

@mnapoli
Copy link
Member

mnapoli commented Feb 12, 2024

Could you add documentation to help understand this? Hard for me to understand right now TBH.

Also good point, we want to avoid BC breaks 👍

@robinlehrmann robinlehrmann force-pushed the transport-name-resolver branch from e59d589 to b95006f Compare February 12, 2024 17:25
@robinlehrmann robinlehrmann force-pushed the transport-name-resolver branch from b95006f to 1fd1f89 Compare February 12, 2024 18:02
@robinlehrmann
Copy link
Contributor Author

robinlehrmann commented Feb 12, 2024

@tyx yes, this would be a BC break :) I added the $transportName again. I also added some tests as well, to make sure both ways (automatic detection and manually set) works as expected.

In an other merge request I would refactor the SnsConsumerTest if you agree and use Prophecy as well to have only one way of creating mocks. But this is for later. Could you take a look again?

@mnapoli Yes, sure. It's in the README.md file now, let me know if this is understandable :)

And by the way should I implement the transport detector for EventBridge as well because I forgot that one? :)
It seams to be not possible to do it with EventBridge, because there is no eventSource or something like this.

@robinlehrmann robinlehrmann force-pushed the transport-name-resolver branch from a2525e5 to 7878370 Compare February 13, 2024 09:10
@robinlehrmann robinlehrmann force-pushed the transport-name-resolver branch from 7878370 to 74111e7 Compare February 13, 2024 10:44
@robinlehrmann robinlehrmann requested a review from tyx February 13, 2024 10:48
@robinlehrmann robinlehrmann requested a review from tyx February 13, 2024 15:02
@robinlehrmann
Copy link
Contributor Author

@tyx @mnapoli when are you planning to merge this? :)

@tyx
Copy link
Contributor

tyx commented Feb 19, 2024

Hi ! I'm just a bref user like you ;)

I'm sure Matthieu will do his best to review as soon as he has some free time

@robinlehrmann
Copy link
Contributor Author

Hello @mnapoli I don't want to pressure, but can you take a look at this, it's quite urgent 😅 would be really nice if this can be merged. Thank you!

@SinaFetrat
Copy link

Hi @mnapoli , i was wondering if there is any update or plan about merging this?
because I also have the same problem and this will help a lot to get rid of many lambdas

@mnapoli
Copy link
Member

mnapoli commented Apr 10, 2024

Merging since we have now 3 different users asking for it. It's always hard to judge how actually used/useful a feature is when not using it, I've made the mistake of merging stuff for 1% of users before and regretted it 😬

Also I have a rule not to merge stuff I don't understand or can't maintain. I'm breaking that rule here, I trust you guys to have reviewed and maintain it if you face issues 😅

@mnapoli mnapoli merged commit 023ed34 into brefphp:master Apr 10, 2024
@robinlehrmann robinlehrmann deleted the transport-name-resolver branch April 10, 2024 13:54
@mnapoli
Copy link
Member

mnapoli commented Apr 10, 2024

I have no idea why but it seems GitHub Actions didn't run on this PR? On master PHPStan fails now, any idea if it's related to this PR?

@robinlehrmann
Copy link
Contributor Author

I will take a look on this :) @mnapoli

@robinlehrmann
Copy link
Contributor Author

@mnapoli I ignored the error here: https://github.com/brefphp/symfony-messenger/pull/85/files is this ok for you?

@Cafeine42
Copy link

Hi @robinlehrmann ,

I'm having trouble understanding the purpose and logic behind the SqsTransportNameResolver class. From what I see, it seems to be trying to match SQS transport names in the form of :
sqs://arn:aws:sqs:eu-west-3:000000000000:project-default-high
However, in our Messenger configuration, the transport names look more like this:
https://sqs.eu-west-3.amazonaws.com/000000000000/project-default-high
These two formats seem fundamentally different — one is an ARN prefixed with sqs://, and the other is a standard HTTPS URL used by the AWS SDK and AWS Async. I'm not sure how or why the resolver is expected to successfully match these two formats.

Could you clarify how we’re supposed to configure the DSNs in order to make the automatic detection work as intended?

Thanks in advance!

@robinlehrmann
Copy link
Contributor Author

Hi @Cafeine42

First of all you can see some details about this topic in #83

There you can see the following example:

framework:
    messenger:
        transports:
            async_event_user_changed: '%env(MESSENGER_EVENT_USER_CHANGED_TRANSPORT_DSN)%'
            async_subscriber_send_mail_user_removed: '%env(MESSENGER_SUBSCRIBER_SEND_MAIL_USER_REMOVED_TRANSPORT_DSN)%'
            async_subscriber_backup_user_changed: '%env(MESSENGER_SUBSCRIBER_BACKUP_USER_CHANGED_TRANSPORT_DSN)%'
            
        routing:
            'App\Event\UserChangedEvent': async_event_user_changed

The "problem" beforehand was, that it was necessary to configure the $transportName in here: https://github.com/brefphp/symfony-messenger/blob/master/src/Service/Sqs/SqsConsumer.php#L38
This ends up in creating one lambda per each consumer, because the consumers are limited to one transport and the environment variables couldn't be configured in lambda on the fly.

In my scenario it was the case that I configured one SNS-Topic and at least one SQS per event. (Every SQS Queue has one subscriber, so if you have a different subscriber I created a different queue, to handle the dead letter queue entries as expected on failures. I would also suggest that for you, because this is also the best practice approach for AWS SNS fanout -> AWS SQS)

However in your case you want to handle more then one event in your transports so I'm not 100% if this is possible with this approach or not. So if not you can just configure the $transportName in your SQSConsumer manually, then you should not effect your case.

@Cafeine42
Copy link

Cafeine42 commented May 9, 2025

In my use case, I use different SQS queues and different transports to manage the priority of exchanged messages. This way, high-priority messages—which are produced in small quantities and are very quick to process—don’t end up buried in a non-priority queue with potentially long processing times.

We use SNS+SQS in another use case, but in this case, we don’t need it.

I will create a PR so that transport resolution works when DSNs are configured as indicated in the Symfony documentation (https://sqs.eu-west-3.amazonaws.com/123456789012/messages).

Here : #98

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Architectural Improvements for SNS-SQS-Lambda Integration with Symfony Messenger

5 participants