Skip to content

modify test scenarios of grub2_argument template to handle variables #12428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
marcusburghardt merged 1 commit into from
Sep 27, 2024
Merged

modify test scenarios of grub2_argument template to handle variables #12428

marcusburghardt merged 1 commit into from
Sep 27, 2024

Conversation

@vojtapolasek
Copy link
Collaborator

@vojtapolasek vojtapolasek commented Sep 25, 2024

Description:

  • ensure that all test scenarios can handle case when the rule uses XCCDF variable and not

Rationale:

  • test scenarios in such cases were not behaving correctly, it did not affect the rule it self

Review Hints:

Build content for rhel8 and rhel9 product.
Then pick two rules templated with grub2_argument template so that one uses xccdf variable (grub2_l1tf_argument) and another does not (grub2_audit_argument).
Run automatus for those rules.

@vojtapolasek
modify test scenarios of grub2_argument template so that cases when X…
ab92fc4 
…CCDF variable is used is are handled properly
@vojtapolasek vojtapolasek added the Test Suite Update in Test Suite. label Sep 25, 2024
@vojtapolasek vojtapolasek added this to the 0.1.75 milestone Sep 25, 2024
@github-actions
Copy link

github-actions bot commented Sep 25, 2024

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment
Open in Gitpod

Oracle Linux 8 Environment
Open in Gitpod

@github-actions
Copy link

github-actions bot commented Sep 25, 2024

🤖 A k8s content image for this PR is available at:
ghcr.io/complianceascode/k8scontent:12428
This image was built from commit: ab92fc4

Click here to see how to deploy it

If you alread have Compliance Operator deployed:
utils/build_ds_container.py -i ghcr.io/complianceascode/k8scontent:12428

Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and:
CONTENT_IMAGE=ghcr.io/complianceascode/k8scontent:12428 make deploy-local

@qlty-cloud-legacy
Copy link

qlty-cloud-legacy bot commented Sep 25, 2024

Code Climate has analyzed commit ab92fc4 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 59.5% (0.0% change).

View more on Code Climate.

@marcusburghardt marcusburghardt self-assigned this Sep 27, 2024
marcusburghardt
marcusburghardt approved these changes Sep 27, 2024
View reviewed changes
Copy link
Member

@marcusburghardt marcusburghardt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested locally and the PR fixed the test scenarios for rules with variables.

@marcusburghardt
Copy link
Member

marcusburghardt commented Sep 27, 2024

Besides the CI tests, I also tested the PR locally in a RHEL 9 VM.

Before the PR:

./tests/automatus.py rule --libvirt qemu:///session rhel9 --datastream build/ssg-rhel9-ds.xml --dontclean --remediate-using bash grub2_l1tf_argument
Setting console output to log level INFO
INFO - The base image option has not been specified, choosing libvirt-based test environment.
WARNING - Script correct_value_mix_entries_and_grubenv.pass.sh is not applicable on given platform
WARNING - Script arg_not_in_grubenv_and_not_referenced.pass.sh is not applicable on given platform
WARNING - Script correct_value_substring_right.pass.sh is not applicable on given platform
WARNING - Script correct_value_etcdefault_dir.pass.sh is not applicable on given platform
WARNING - Script arg_not_in_grubenv_but_referenced.fail.sh is not applicable on given platform
WARNING - Script correct_value_etcdefault_dir_noupdate.fail.sh is not applicable on given platform
WARNING - Script correct_value_substring_left.pass.sh is not applicable on given platform
WARNING - Script wrong_value_grubenv.fail.sh is not applicable on given platform
WARNING - Script correct_value_grubenv_only.pass.sh is not applicable on given platform
WARNING - Script wrong_value_etcdefault_dir.fail.sh is not applicable on given platform
INFO - xccdf_org.ssgproject.content_rule_grub2_l1tf_argument
INFO - Script correct_value_noupdate.fail.sh using profile (all) OK
INFO - Script arg_not_in_entries.fail.sh using profile (all) OK
ERROR - Script correct_recovery_disabled.pass.sh using profile (all) found issue:
ERROR - Rule evaluation resulted in fail, instead of expected pass during initial stage 
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_grub2_l1tf_argument'.
INFO - Script wrong_value_etcdefault.fail.sh using profile (all) OK
INFO - Script wrong_value_etcdefaultgrub_recovery_disabled.fail.sh using profile (all) OK
INFO - Script arg_not_in_etcdefaultgrub.fail.sh using profile (all) OK
INFO - Script wrong_value_entries.fail.sh using profile (all) OK
INFO - Script invalid_rescue.pass.sh using profile (all) OK
INFO - Script arg_not_in_etcdefaultgrub_recovery_disabled.fail.sh using profile (all) OK
INFO - Script correct_value_remediated.pass.sh using profile (all) OK

After the PR:

./tests/automatus.py rule --libvirt qemu:///session rhel9 --datastream build/ssg-rhel9-ds.xml --dontclean --remediate-using bash grub2_l1tf_argument
Setting console output to log level INFO
INFO - The base image option has not been specified, choosing libvirt-based test environment.
WARNING - Script wrong_value_grubenv.fail.sh is not applicable on given platform
WARNING - Script correct_value_mix_entries_and_grubenv.pass.sh is not applicable on given platform
WARNING - Script correct_value_substring_right.pass.sh is not applicable on given platform
WARNING - Script correct_value_substring_left.pass.sh is not applicable on given platform
WARNING - Script correct_value_etcdefault_dir.pass.sh is not applicable on given platform
WARNING - Script arg_not_in_grubenv_but_referenced.fail.sh is not applicable on given platform
WARNING - Script wrong_value_etcdefault_dir.fail.sh is not applicable on given platform
WARNING - Script correct_value_etcdefault_dir_noupdate.fail.sh is not applicable on given platform
WARNING - Script arg_not_in_grubenv_and_not_referenced.pass.sh is not applicable on given platform
WARNING - Script correct_value_grubenv_only.pass.sh is not applicable on given platform
INFO - xccdf_org.ssgproject.content_rule_grub2_l1tf_argument
INFO - Script invalid_rescue.pass.sh using profile (all) OK
INFO - Script wrong_value_etcdefaultgrub_recovery_disabled.fail.sh using profile (all) OK
INFO - Script arg_not_in_entries.fail.sh using profile (all) OK
INFO - Script wrong_value_etcdefault.fail.sh using profile (all) OK
INFO - Script correct_value_remediated.pass.sh using profile (all) OK
INFO - Script wrong_value_entries.fail.sh using profile (all) OK
INFO - Script correct_recovery_disabled.pass.sh using profile (all) OK
INFO - Script arg_not_in_etcdefaultgrub_recovery_disabled.fail.sh using profile (all) OK
INFO - Script correct_value_noupdate.fail.sh using profile (all) OK
INFO - Script arg_not_in_etcdefaultgrub.fail.sh using profile (all) OK

@marcusburghardt marcusburghardt added the productization-issue Issue found in upstream stabilization process. label Sep 27, 2024
@marcusburghardt marcusburghardt merged commit 7af6c1c into ComplianceAsCode:master Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@marcusburghardt marcusburghardt marcusburghardt approved these changes

@jan-cerny jan-cerny Awaiting requested review from jan-cerny jan-cerny is a code owner

Assignees

@marcusburghardt marcusburghardt

Labels

productization-issue Issue found in upstream stabilization process. Test Suite Update in Test Suite.

Projects

None yet

Milestone

0.1.75

Development

Successfully merging this pull request may close these issues.

2 participants

@vojtapolasek @marcusburghardt