Install Bundled Gems in Deployment Mode #66536

Hamms · 2025-06-13T21:17:32Z

Follow-up to #65533, which upgraded us not only to Ruby 3.1 but also to Bundler 2.5.17 from v2.3.

In v2.4, Bundler removed the "auto-sudo" feature. So, hooray for less-potentially-dangerous defaults! Unfortunately, we have indeed been violating Bundler's recommendations by installing gems as root, and were unknowingly relying on the auto-sudo behavior to allow the deploy-storybook script to work.

This PR proposes that we stop installing our gems as root. Specifically, it switches us to use the offically-recommended deployment option on our persistent managed servers, and switches the necessary build steps to use the local ubuntu user rather than executing as root. The result is that all of our gems, their dependencies, and the associated executables will be installed to the vendor/bundle directory at the root of the project.

This does mean we will need to consistently use bundle exec in all of our build scripts in order to be able to discover the executables since they are no longer being installed to a $PATH directory, but fortunately as far as I've been able to tell we already do so.

Links

Testing story

Tested on adhocs. I verified both that we are able to successfully provision a new one based on this code, and also that an existing adhoc provisioned based on staging will successfully upgrade when this code is merged in.

Deployment strategy

New chef-managed servers will be configured to use deployment mode on startup; existing chef-managed servers will have their configuration updated by the build. We could follow this PR up with one which removes the second one, but I don't mind a bit of redundancy here. Open to feedback if you think otherwise!

Follow-up work

As a follow up, we will also uninstall all root-installed gems from all of our persistent managed servers.

Also, we can reenable the storybook build.

… non-root user

…servers

This reverts commit cb27ae8.

This reverts commit 99c7aba.

…we're able to persist it

…cken-egg problem with updating the config

…ther than ruby

Hamms · 2025-07-02T01:06:15Z

cookbooks/cdo-apps/recipes/bundle_bootstrap.rb

+  # Temporarily disable the conditional check so that we always run bundle
+  # install on every build. This will help us avoid the chicken-egg problem we
+  # otherwise get when we try to rely on a Ruby-initiated process to switch
+  # from running bundle install as root to running as local user.


Specifically, I was getting the following error:

Installing �[1mdashboard�[22m bundle... bundle config set --local without development production staging test levelbuilder integration bundle config set --local deployment 'true' bundle install --quiet --jobs 8 --deployment rake aborted! 'bundle install --quiet --jobs 8 --deployment' returned 1 [GEM]/bundler-2.5.17/lib/bundler/source/git.rb:233:in `rescue in load_spec_files': https://github.com/code-dot-org/sprockets.git (at concurrent_asset_bundle_3.x@35caa45) is not yet checked out. Run `bundle install` first. (Bundler::GitError) from [GEM]/bundler-2.5.17/lib/bundler/source/git.rb:229:in `load_spec_files' from [GEM]/bundler-2.5.17/lib/bundler/source/path.rb:95:in `local_specs' from [GEM]/bundler-2.5.17/lib/bundler/source/git.rb:198:in `specs' from [GEM]/bundler-2.5.17/lib/bundler/lazy_specification.rb:98:in `materialize_for_installation' from [GEM]/bundler-2.5.17/lib/bundler/gem_helpers.rb:66:in `map' from [GEM]/bundler-2.5.17/lib/bundler/gem_helpers.rb:66:in `select_best_local_platform_match' from [GEM]/bundler-2.5.17/lib/bundler/spec_set.rb:283:in `specs_for_dependency' from [GEM]/bundler-2.5.17/lib/bundler/spec_set.rb:34:in `block in for' from [GEM]/bundler-2.5.17/lib/bundler/spec_set.rb:22:in `loop' from [GEM]/bundler-2.5.17/lib/bundler/spec_set.rb:22:in `for' from [GEM]/bundler-2.5.17/lib/bundler/spec_set.rb:115:in `materialize' from [GEM]/bundler-2.5.17/lib/bundler/definition.rb:584:in `materialize' from [GEM]/bundler-2.5.17/lib/bundler/definition.rb:193:in `specs' from [GEM]/bundler-2.5.17/lib/bundler/definition.rb:259:in `specs_for' from [GEM]/bundler-2.5.17/lib/bundler/runtime.rb:18:in `setup' from [GEM]/bundler-2.5.17/lib/bundler.rb:164:in `setup' from [GEM]/bundler-2.5.17/lib/bundler/setup.rb:32:in `block in <top (required)>' from [GEM]/bundler-2.5.17/lib/bundler/ui/shell.rb:159:in `with_level' from [GEM]/bundler-2.5.17/lib/bundler/ui/shell.rb:111:in `silence' from [GEM]/bundler-2.5.17/lib/bundler/setup.rb:32:in `<top (required)>' from <internal:/usr/local/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require' from <internal:/usr/local/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require' [GEM]/bundler-2.5.17/lib/bundler/source/path.rb:197:in `load_spec_files': The path `[CDO]/vendor/bundle/ruby/3.1.0/bundler/gems/sprockets-35caa45a78b7` does not exist. (Bundler::PathError) from [GEM]/bundler-2.5.17/lib/bundler/source/git.rb:230:in `load_spec_files' from [GEM]/bundler-2.5.17/lib/bundler/source/path.rb:95:in `local_specs' from [GEM]/bundler-2.5.17/lib/bundler/source/git.rb:198:in `specs' from [GEM]/bundler-2.5.17/lib/bundler/lazy_specification.rb:98:in `materialize_for_installation' from [GEM]/bundler-2.5.17/lib/bundler/gem_helpers.rb:66:in `map' from [GEM]/bundler-2.5.17/lib/bundler/gem_helpers.rb:66:in `select_best_local_platform_match' from [GEM]/bundler-2.5.17/lib/bundler/spec_set.rb:283:in `specs_for_dependency' from [GEM]/bundler-2.5.17/lib/bundler/spec_set.rb:34:in `block in for' from [GEM]/bundler-2.5.17/lib/bundler/spec_set.rb:22:in `loop' from [GEM]/bundler-2.5.17/lib/bundler/spec_set.rb:22:in `for' from [GEM]/bundler-2.5.17/lib/bundler/spec_set.rb:115:in `materialize' from [GEM]/bundler-2.5.17/lib/bundler/definition.rb:584:in `materialize' from [GEM]/bundler-2.5.17/lib/bundler/definition.rb:193:in `specs' from [GEM]/bundler-2.5.17/lib/bundler/definition.rb:259:in `specs_for' from [GEM]/bundler-2.5.17/lib/bundler/runtime.rb:18:in `setup' from [GEM]/bundler-2.5.17/lib/bundler.rb:164:in `setup' from [GEM]/bundler-2.5.17/lib/bundler/setup.rb:32:in `block in <top (required)>' from [GEM]/bundler-2.5.17/lib/bundler/ui/shell.rb:159:in `with_level' from [GEM]/bundler-2.5.17/lib/bundler/ui/shell.rb:111:in `silence' from [GEM]/bundler-2.5.17/lib/bundler/setup.rb:32:in `<top (required)>' from <internal:/usr/local/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require' from <internal:/usr/local/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require' Tasks: TOP => build:all => build:dashboard

I have not been able to figure out why bundle install would fail with complaints about missing packages, and recommend that I run bundle install to resolve. I also have not been able to reproduce the problem manually; all of my attempt to run bundle install via remote shell succeed, all of my attempts to do so via build.rake fail, and do so in a way that prevents subsequent builds from running.

Running it via chef seems like a reasonable compromise. I'll plan to merge #66856 as a follow up once all servers are updated.

Follow-up to #66536

Hamms · 2025-07-02T01:12:45Z

cookbooks/Berksfile

  cookbooks = Chef::ServerAPI.new(Chef::Config[:chef_server_url]).get('/cookbooks')
-  cookbooks.each_key do |path|
+  cookbooks.keys.select do |path|
+    path.start_with? 'cdo-'


This change is necessary for us to avoid attempting to treat the vendor directory produced by bundle install as a cookbook. It does commit us to always prefixing all of our cookbooks with cdo-, but as that is the case and has been for quite some time, I feel comfortable with that.

Open to suggestions if you have ideas for a better way to handle this!

Hamms · 2025-07-02T01:15:26Z

.gitignore

+/vendor/bundle
+/cookbooks/vendor/bundle


These two are the only new gitignore entries

sureshc

I don't fully understand all of the mechanics with Chef, but the testing approach (provision a new adhoc, as well as deployed to an existing adhoc) gives me confidence that the risks with this change are low.

Follow-up to #66536

This reverts commit 0eb4534.

…7115)" This reverts commit 9a7e95f.

Follow-up to #67155, which included a change to `cookbooks/Berksfile` to only treat subdirectories starting with `cdo-` as cookbooks (see [this comment](#66536 (review)) for details). Unfortunately, because the `update_cookbook_versions` script only executes in the staging environment I missed it in my testing. This PR represents a quick fix-forward to bring that script in line with our updated logic.

* add chef script to smooth out the transition to bundle deployment mode * ignore new vendor/ directory everywhere it will now appear * include all generated vendor dirs * include bundle install from project dir in the transition script * Revert "Revert "Install Bundled Gems in Deployment Mode (#66536)" (#67115)" This reverts commit 9a7e95f. * stop unsetting deployment mode now that we're ready to use it

Follow-up to #66536

Hamms added 4 commits June 10, 2025 17:06

completely remove bundler_use_sudo option

dbf003e

configure bundle to use deployment mode and execute bundle install as…

893ddd1

… non-root user

gitignore the directory into which gems will be installed on managed …

5dbbb64

…servers

temporarily comment out some seeding logic that breaks adhocs

99c7aba

Hamms mentioned this pull request Jun 14, 2025

test bundle deployment mode #66540

Closed

8 tasks

Hamms added 10 commits June 17, 2025 12:38

install bundle executables to ~/bin

cb27ae8

Revert "install bundle executables to ~/bin"

89e4cfc

This reverts commit cb27ae8.

install gem executables into PATH-accessible directory

bc720c8

Merge branch 'staging' into elijah/bundle-deployment-mode

980983f

let's try to make the default bin dir work

7839829

Merge branch 'staging' into elijah/bundle-deployment-mode

53e9025

revert some unnecessary changes

d8576df

empty commit to test adhoc rebuild

0be8c37

use deployment mode on adhocs

f00a318

fix 'root running as normal user' implementation

e01cffa

Hamms changed the title ~~Install Bundles in Deployment Mode~~ Install Bundled Gems in Deployment Mode Jun 23, 2025

Hamms added the Ruby Update Everything related to work to update the version of Ruby our codebase runs on label Jun 23, 2025

Hamms added 7 commits June 26, 2025 12:55

ignore the new cookbooks/vendor directory

4092d63

Merge branch 'staging' into elijah/bundle-deployment-mode

40a4f86

Revert "temporarily comment out some seeding logic that breaks adhocs"

28845d7

This reverts commit 99c7aba.

use deprecated deployment flag to force the mode in the build before …

f280357

…we're able to persist it

exclusively use deprecated flag for initial install, to avoid the chi…

1f8706e

…cken-egg problem with updating the config

temporarily always run bundle install as part of chef

3d7db18

revert to long-term pattern; we'll rely on chef for the transition ra…

7f61d9a

…ther than ruby

Hamms commented Jul 2, 2025

View reviewed changes

Hamms added a commit that referenced this pull request Jul 2, 2025

Restore bundle install Conditional

cf5a3c7

Follow-up to #66536

Hamms mentioned this pull request Jul 2, 2025

Restore bundle install Conditional #66856

Merged

Hamms commented Jul 2, 2025

View reviewed changes

.gitignore

Comment on lines +41 to +42

/vendor/bundle

/cookbooks/vendor/bundle

Copy link

Contributor Author

Hamms Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two are the only new gitignore entries

Hamms mentioned this pull request Jul 2, 2025

Re-own .bundle Directories on Servers #66833

Merged

Hamms marked this pull request as ready for review July 3, 2025 18:30

Hamms requested a review from a team as a code owner July 3, 2025 18:30

sureshc approved these changes Jul 7, 2025

View reviewed changes

Hamms mentioned this pull request Jul 8, 2025

Remove Old Root-Installed Gems #66922

Merged

Hamms merged commit 0eb4534 into staging Jul 15, 2025
12 checks passed

Hamms deleted the elijah/bundle-deployment-mode branch July 15, 2025 18:25

Hamms added a commit that referenced this pull request Jul 15, 2025

Restore bundle install Conditional

8193ed2

Follow-up to #66536

Hamms added a commit that referenced this pull request Jul 15, 2025

Revert "Install Bundled Gems in Deployment Mode (#66536)"

c1fc322

This reverts commit 0eb4534.

Hamms mentioned this pull request Jul 15, 2025

Revert "Install Bundled Gems in Deployment Mode" #67115

Merged

Hamms added a commit that referenced this pull request Jul 15, 2025

Revert "Install Bundled Gems in Deployment Mode (#66536)" (#67115)

9a7e95f

This reverts commit 0eb4534.

Hamms added a commit that referenced this pull request Jul 15, 2025

Revert "Revert "Install Bundled Gems in Deployment Mode (#66536)" (#6…

c68f8f3

…7115)" This reverts commit 9a7e95f.

Hamms mentioned this pull request Jul 15, 2025

Revert "Revert "Install Bundled Gems in Deployment Mode"" #67117

Merged

Hamms added a commit that referenced this pull request Jul 23, 2025

Revert "Revert "Install Bundled Gems in Deployment Mode (#66536)" (#6…

7c9b77a

…7115)" This reverts commit 9a7e95f.

Hamms mentioned this pull request Jul 23, 2025

Install Gems in Bundle Deployment Mode Without Switching Mode #67155

Merged

Hamms mentioned this pull request Jul 31, 2025

Fix update_cookbook_versions Script for new Bundle Install #67464

Merged

Hamms mentioned this pull request Aug 11, 2025

Revert "Revert "Revert "Revert "Install Bundled Gems in Deployment Mode"""" #67642

Merged

Hamms added a commit that referenced this pull request Sep 5, 2025

Restore bundle install Conditional (#66856)

0791e69

Follow-up to #66536

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Install Bundled Gems in Deployment Mode #66536

Install Bundled Gems in Deployment Mode #66536

Uh oh!

Hamms commented Jun 13, 2025 •

edited

Loading

Uh oh!

Hamms Jul 2, 2025 •

edited

Loading

Uh oh!

Hamms Jul 2, 2025 •

edited

Loading

Uh oh!

Hamms Jul 2, 2025

Uh oh!

sureshc left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Install Bundled Gems in Deployment Mode #66536

Install Bundled Gems in Deployment Mode #66536

Uh oh!

Conversation

Hamms commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Links

Testing story

Deployment strategy

Follow-up work

Uh oh!

Hamms Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hamms Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hamms Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

sureshc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Hamms commented Jun 13, 2025 •

edited

Loading

Hamms Jul 2, 2025 •

edited

Loading

Hamms Jul 2, 2025 •

edited

Loading