mirrors/bitcoin
1
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2025-02-20 14:05:23 +01:00
Code Issues Projects Releases Wiki Activity

Merge bitcoin/bitcoin#30872: test: fix exclude parsing for functional runner

Browse source 
72b46f28bf test: fix exclude parsing for functional runner (Max Edwards)

Pull request description:

  This restores previous behaviour of being able to exclude a test by name without having to specify .py extension.

  It was noticed in https://github.com/bitcoin/bitcoin/issues/30851 that tests were no longer being excluded.

  PR https://github.com/bitcoin/bitcoin/pull/30244 introduced being able to exclude a specific tests based on args (such as `--exclude "rpc_bind.py --ipv6`) but it made the wrong assumption that test names intended to be excluded would include the .py extension.

  The following https://github.com/bitcoin/bitcoin/pull/30244#issuecomment-2344009687 shows that this is not how the `--exclude` flag was used in CI.

  https://github.com/bitcoin/bitcoin/pull/30244#issuecomment-2344009687 gave three examples of `--exclude` being used in CI so I compared the number of tests that the runner would run for these three examples in three situations, before #30244 was introduced, in master today and with this PR applied.

  Example:

  `--previous-releases --coverage --extended --exclude feature_dbcrash`

  Test count:
  Before #30244 introduced: 314
  Master: 315
  With this PR: 314

  Example:

  `--exclude feature_init,rpc_bind,feature_bind_extra`

  Test count:
  Before #30244 introduced: 306
  Master 311
  With this PR: 306

  Example:

  `--exclude rpc_bind,feature_bind_extra`

  Before #30244 introduced:  307
  Master 311
  With this PR: 307

  I've also tested that the functionality introduced with #30244 remains and we can still exclude specific tests by argument.

ACKs for top commit:
  maflcko:
    review ACK 72b46f28bf
  willcl-ark:
    ACK 72b46f28bf

Tree-SHA512: 37c0e3115f4e3efdf9705f4ff8cd86a5cc906aacc1ab26b0f767f5fb6a953034332b29b0667073f8382a48a2fe9d649b7e60493daf04061260adaa421419d8c8
This commit is contained in:
merge-script 2024-09-12 15:30:34 +01:00
commit 7d43bca052
No known key found for this signature in database
GPG key ID: 2EEB9F5CC09526C1
1 changed files with 12 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

23
test/functional/test_runner.py
View file

@ -522,23 +522,24 @@ def main():
test_list += BASE_SCRIPTS
# Remove the test cases that the user has explicitly asked to exclude.
# The user can specify a test case with or without the .py extension.
if args.exclude:
def print_warning_missing_test(test_name):
print("{}WARNING!{} Test '{}' not found in current test list.".format(BOLD[1], BOLD[0], test_name))
def remove_tests(exclude_list):
if not exclude_list:
print_warning_missing_test(exclude_test)
for exclude_item in exclude_list:
test_list.remove(exclude_item)
exclude_tests = [test.strip() for test in args.exclude.split(",")]
for exclude_test in exclude_tests:
if exclude_test.endswith('.py'):
# Remove <test_name>.py and <test_name>.py --arg from the test list
exclude_list = [test for test in test_list if test.split('.py')[0] == exclude_test.split('.py')[0]]
if not exclude_list:
print_warning_missing_test(exclude_test)
for exclude_item in exclude_list:
test_list.remove(exclude_item)
# A space in the name indicates it has arguments such as "wallet_basic.py --descriptors"
if ' ' in exclude_test:
remove_tests([test for test in test_list if test.replace('.py', '') == exclude_test.replace('.py', '')])
else:
try:
test_list.remove(exclude_test)
except ValueError:
print_warning_missing_test(exclude_test)
# Exclude all variants of a test
remove_tests([test for test in test_list if test.split('.py')[0] == exclude_test.split('.py')[0]])
if args.filter:
test_list = list(filter(re.compile(args.filter).search, test_list))