Switch to Amazon Linux 2023 for Bref v3 #122

mnapoli · 2023-09-18T12:14:32Z

Bref needs to support AL2023, this PR will be included in Bref v3.

It removes a lot of custom system libraries we compile because AL2023 contains more recent versions: https://docs.aws.amazon.com/linux/al2023/release-notes/all-packages-AL2023.1.html

php-80/Dockerfile

mnapoli · 2023-09-18T12:15:51Z

php-81/Dockerfile

-#   - curl
-#   - php
-RUN yum install -y perl-IPC-Cmd
-ENV VERSION_OPENSSL=3.0.10


Native version: 3.0.8

Security issues in the native version?

I tried asking around but have no answers yet. Do you know if the versions compiled in AL might have some security fixes applied?

I'm just surprised AL would ship with security issues, how do all customers do? I wouldn't expect everyone to compile everything from source, right?

We backport security fixes to the OpenSSL versions that we ship. See https://alas.aws.amazon.com/alas2023.html for updates.

For example: https://alas.aws.amazon.com/AL2023/ALAS-2023-306.html and https://alas.aws.amazon.com/AL2023/ALAS-2023-222.html are some recent ones.

If there's anything from newer OpenSSL that you need, please do let us know and submit an issue to https://github.com/amazonlinux/amazon-linux-2023/ for us to look at.

We need a version that supports quic possibly, for http/3 in curl, also. I am not sure how long it takes AWS to apply security fixes. I like the flexibility that keeping this gives us to easily re-compile, or for me to fork and re-compile to get something patched immediately.

php-81/Dockerfile

mnapoli · 2023-09-18T12:17:08Z

php-81/Dockerfile

-# #   - libnghttp2
-# # Needed by:
-# #   - php
-ENV VERSION_CURL=8.3.0


Native version: 8.2.1

This won't be possible, once we need to start building curl with http/3 enabled.

Note that we do move curl forward in Amazon Linux fairly regularly, often as part of patching security issues.

If there's anything specifically not enabled in our curl build that you need, please file an issue over at https://github.com/amazonlinux/amazon-linux-2023/issues so we can look into it.

Note that in AL2023 there is both a curl and curl-minimal package (and the associated libcurl and libcurl-minimal). The minimal ones are installed by default, and we try and capture the 99% of use cases there (e.g. don't build in ldap:// support and the like).

php-81/Dockerfile

GrahamCampbell · 2023-09-18T12:49:12Z

AL2023 is eventually going to happen

I am actually not so sure. 😆

mnapoli · 2023-09-27T14:42:33Z

@GrahamCampbell that wasn't long :) aws/aws-lambda-base-images#92

GrahamCampbell · 2023-09-27T14:53:10Z

Ha, it's not available on the Lambda APIs and cloudformation yet. ;)

stewartsmith · 2023-10-09T18:05:47Z

php-80/Dockerfile

-
 ###############################################################################
 # LIBSODIUM
 # https://github.com/jedisct1/libsodium/releases


Building libsodium shouldn't be required on AL2023 as there's a version in the repositories as of AL2023.2 (see https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.2.20230920.html )

Again, it is not the latest version.

php-81/Dockerfile

mnapoli · 2023-10-10T08:56:45Z

Thanks so much for jumping in the discussion @stewartsmith.

There is some balance that we need to find.

Would it be reasonable to apply the following logic:

When we need specific features (random example: HTTP3 support in curl) that are not present in AL2023, we recompile the libraries
The rest of the time, we rely on AL2023 versions for CVEs and security fixes

Is that reasonable enough? Can we assume that all CVEs are patched in AL2023 eventually? (I understand it may take a few hours/days depending on the issue)

The benefits of that approach would be:

our compilation scripts are simpler
our layers might potentially be lighter (faster cold starts)
security fixes are delivered in the Lambda OS, not via a new version of the Bref layers -> for most Bref users, that means that security fixes are actually applied faster, because most Bref users don't necessarily update Bref layers every week or so (meaning they are actually using older unpatched system libraries provided by Bref).

WDYT?

stewartsmith · 2023-10-12T14:50:52Z

Building anything from source and replacing an OS component with it carries various risks. Naturally it means you have to keep it updated, but there's also the nuance of building it with the same hardening and optimisations as the one in the OS, and ensuring ABI compatibility with the one in the OS. It's not something we recommend doing.

It is (of course) something which is possible to do, as it's basically what we do all the time when building updates for the OS. From a "does the Amazon Linux team support this" standpoint, replacing core system libraries falls into "can you reproduce the issue without that change" territory - i.e. a freedom that Free and Open Source Software gives you, but the vendor doesn't necessarily provide support for if you pick up the (metaphorical or literal) phone.

I would recommend building RPMs instead of manually from source tarballs though, as that way you capture all the hardening and optimization flags that are added in the RPM macros.

In Amazon Linux, security is job zero. We do fix security issues of all severity, and aim to do so not just in the most recent major version of the OS. The places where don't patch something basically boils down to risk outweighing the benefit - such as Low CVEs that are a quite invasive fix that we can't reasonably bring in without creating other, more severe issues.

tests/test_2_extensions.php

mnapoli · 2023-10-17T14:23:01Z

Thanks a lot @stewartsmith for chiming in. I've taken some time to think about it, and also discuss this with a few Bref users. For example I discussed this topic with a bank running on Bref (i.e. they have huge security constraints).

The feedback was quite simple: if we build our own versions, we (Bref project) are responsible for the security related to those libraries. If we don't, AWS is.

To summarize:

Bref users looking for strong security guarantees will usually prefer to delegate as much as possible to AWS (https://docs.aws.amazon.com/whitepapers/latest/security-overview-aws-lambda/the-shared-responsibility-model.html). This allows ticking a lot of boxes.
Additionally, most Bref users will not update the runtime versions frequently enough, leaving them more exposed to CVEs and security issues (in the case where we build libraries). By using system libraries instead, AWS is able to patch the underlying runtime automatically, which gives a more secure environment.

That leads me to conclude that we should aim to compile as few libraries as possible. This will benefit both extremes of the spectrum: users with strong security requirements and those that don't update frequently (i.e. care less about security, if I can say so).

The downside is that we have no way to control the frequency/speed at which AWS patches security issues. This basically means putting more trust into AWS over Bref regarding security, and I think that is a reasonable choice 😅

This PR is still WIP, but I'll be slowly updating it to reflect that. I.e. the build scripts should be simpler with AL2023 than AL2.

Also since we'll want to support both, I'll probably reorganize the code entirely. We will also likely not support PHP 8.0 with AL2023.

GrahamCampbell · 2023-10-17T22:34:05Z

php-80/Dockerfile

 # AWS provides a Docker image that we use here:
 # https://github.com/amazonlinux/container-images/tree/amzn2
-FROM public.ecr.aws/lambda/provided:al2-${IMAGE_VERSION_SUFFIX} as build-environment
+FROM public.ecr.aws/lambda/provided:al2023-preview-${IMAGE_VERSION_SUFFIX} as build-environment


is there a way to pin this to al2023.2, rather than just al2023?

hmmm, not really. just al2023-preview.2023.10.17.17

shiki-sou · 2024-05-15T03:05:32Z

Hello,
Thank you for your continuous work on the Bref project.
It seems this ticket has been postponed for a little bit while.
I'm interested in how long Amazon Linux 2023 will be supported.
And is there any key milestones or timelines related to this support?

hrzbrg · 2024-05-15T05:13:35Z

Amazon Linux 2023 will be supported officially until 2027. However every 2 years Amazon will release a new major version. So in 2025 there will be the release of Amazon Linux 2025, which in turn will be supported until 2029 etc.
See their release cadence: https://docs.aws.amazon.com/linux/al2023/ug/release-cadence.html

shiki-sou · 2024-05-15T06:14:43Z

Amazon Linux 2023 will be supported officially until 2027. However every 2 years Amazon will release a new major version. So in 2025 there will be the release of Amazon Linux 2025, which in turn will be supported until 2029 etc. See their release cadence: https://docs.aws.amazon.com/linux/al2023/ug/release-cadence.html

Thank you for answering my question.
Sorry about not asking question clearly.

I want to know the schedual that brefphp/aws-lambda-layers support Amazon Linux 2023 which this ticket was working on (If I'm not misunderstanding).
Because I found that that brefphp/aws-lambda-layers only suppot amazon linux 2(provided.al2).
See the capture below (I get this capture from Lambda function which I deployed)

mnapoli · 2024-05-17T12:08:39Z

Hi! I have made progress on this offline (not pushed here). This is definitely in my TODO list, though not the most urgent because AL2 works well and is supported by Amazon. But this is not something I have forgotten about or abandoned 👍

shiki-sou · 2024-05-20T06:27:08Z

Hi! I have made progress on this offline (not pushed here). This is definitely in my TODO list, though not the most urgent because AL2 works well and is supported by Amazon. But this is not something I have forgotten about or abandoned 👍

Thank you for your reply.

kevincerro · 2024-06-28T13:00:26Z

Hi!! 🚀

Happy to help with this.
There is something to do? @mnapoli

GrahamCampbell

We probably want newer sqlite, from source.

mnapoli · 2024-10-24T07:31:37Z

@GrahamCampbell why? Are we missing a feature with the version shipped in AL2023?

# Conflicts: # layers/fpm-dev/Dockerfile # php-83/Dockerfile

GrahamCampbell · 2025-05-24T21:35:39Z

@GrahamCampbell why? Are we missing a feature with the version shipped in AL2023?

Yes. They ship a very old version in AL2023 too.

…ravel 11

mnapoli · 2025-05-25T14:34:02Z

@GrahamCampbell what feature are we missing? I just added a test to make sure we have the version that is recent enough for Laravel 11, it seems we're good on this.

GrahamCampbell · 2025-05-25T15:34:27Z

I think the version in Amazon Linux 2023 is high enough for Laravel 11, but not for Drupal 10. We could ask for AWS to upgrade the sqlite version I guess? It's also not clear what version of AL 2023 lambda actually uses. It would be nice if we could control the minor version.

mnapoli · 2025-05-26T09:26:39Z

The included version is 3.40.0, which is enough for Drupal 10, but it seems to not be good for Drupal 11 which requires 3.45:

https://www.drupal.org/docs/getting-started/system-requirements/database-server-requirements

I've opened an issue to request a package upgrade: amazonlinux/amazon-linux-2023#961 Let's see where that gets us.

# Conflicts: # php-83/Dockerfile

GrahamCampbell · 2025-08-15T14:20:05Z

Looks like we still need to install sqlite ourselves.

That will help know which versions are installed in Bref, and decide which ones to compile or not.

mnapoli · 2025-08-15T14:56:32Z

👍 that's done

FYI I pushed bca39a9 so that we have a reference of the package versions in AL2023. That will be much easier to check which version is available by default and decide if we want to compile a package or not.

mnapoli · 2025-08-20T18:43:23Z

@GrahamCampbell I'm going to merge so that we can keep v3 moving forward (e.g. I can do more tests and benchmarks).

The discussion on compiling some dependencies is still open if there are good reasons to do so (e.g. support a specific feature). Let's do this in dedicated PRs on the v3 branch.