Commit Graph

775 Commits

Author SHA1 Message Date
MarcoFalke aaaa432603
ci: Remove "default" test env
It is unclear what the point is of maintaining a "default", the meaning
of which is unclear.
2023-05-29 09:16:21 +02:00
MarcoFalke fa7a87bc7c
ci: Add missing set -e to 01_base_install.sh
Also, set -x for easier debugging.

Also, do the same for ci/test/00_setup_env.sh
2023-05-29 09:16:10 +02:00
Hennadii Stepanov 1f97572b9c
Fix `#include`s in `src/wallet` 2023-05-25 15:52:08 +01:00
willcl-ark 59c8944749
build: disable boost multi index safe mode
Disable boost multi index safe mode by default when configuring with
--enable-debug.

This option can cause transactions to take a long time to be accepted
into the mempool under certain conditions; iterator destruction takes
O(n) time vs O(1) as they are stored in a singly linked list. See
27586 for more information.

Re-enable it on the CI builds which previously had it enabled.

Re-enable it on the msan fuzz target so that we have fuzz tasks testing
with it enabeld and disabled in this repo.
2023-05-23 13:44:07 +01:00
fanquake 456701420b
Merge bitcoin/bitcoin#27672: fuzz: Print error message when FUZZ is missing
fa1b3abc83 ci: Log qa-assets repo last commit (MarcoFalke)
fa22966f33 fuzz: Print error message when FUZZ is missing (MarcoFalke)

Pull request description:

  Some trivial UX improvements.

  * Change the exit code for `PRINT_ALL_FUZZ_TARGETS_AND_ABORT` and `WRITE_ALL_FUZZ_TARGETS_AND_ABORT` to `EXIT_SUCCESS` instead of `Aborted (core dumped)`.
  * Print readable error message when `FUZZ` is missing instead of `Aborted (core dumped)`.
  * Clarify that a fuzz target needs to be compiled into the executable.

ACKs for top commit:
  dergoegge:
    ACK fa1b3abc83

Tree-SHA512: 065ef8920449c64b3516f89a61cb397b505eccf531318c4f3830895d5ff6cd7ae2525cb857320481e3d0ed0b2f8a522cd8f7835e69f021241b6ec297a6102fc8
2023-05-22 12:55:18 +01:00
fanquake 09351f51d2
Merge bitcoin/bitcoin#27699: random: drop syscall wrapper usage for getrandom()
5228223e1f ci: remove MSAN getrandom syscall workaround (fanquake)
d5e06919db random: switch to using getrandom() directly (fanquake)
c2ba3f5b0c random: add [[maybe_unused]] to GetDevURandom (fanquake)
c13c97dbf8 random: getentropy on macOS does not need unistd.h (fanquake)

Pull request description:

  This requires a linux kernel of `3.17`+, which seems entirely
  reasonable. `3.17` went EOL in 2015, and the last supported `3.x` kernel
  (`3.16`) went EOL > 4 years ago, in 2020. For reference, the current
  oldest maintained kernel is `4.14` (released 2017, going EOL Jan 2024).

  Support for `getrandom()` (and `getentropy()`) was added to
  glibc `2.25` https://sourceware.org/legacy-ml/libc-alpha/2017-02/msg00079.html:
  > * The getentropy and getrandom functions, and the <sys/random.h> header
    file have been added.

  and we already require `2.27` or later.

  All that being said, I don't think you would encounter a current day (+~6 months from now)
  system, running with kernel headers older than 3.17 (released 2014) but also having a
  glibc of 2.27+ (released 2018)?

  Removing this (our only) use of `syscall()` also means we can drop a workaround in our MSAN jobs.
  If this is merged, I'll drop the [same workaround in oss-fuzz](25946a5448/projects/bitcoin-core/build.sh (L49-L56)).

ACKs for top commit:
  josibake:
    ACK 5228223e1f
  hebasto:
    ACK 5228223e1f, I've tested build system changes on Ubuntu 22.04 and macOS Monterey 12.6.6 (x86_64).

Tree-SHA512: cc978e08510c461b875ca8c08ae176b4519fa1108f0efd74dcb7474518945357e0184e54423282c9a496de195e4ddc3e221ee78623bd63e24c50cc86acdf32e2
2023-05-22 11:34:58 +01:00
fanquake f998eb7662
Merge bitcoin/bitcoin#27683: ci: remove `RUN_SECURITY_TESTS`
6a936580d1 ci: remove RUN_SECURITY_TESTS (fanquake)

Pull request description:

  We no-longer run any security/symbol checks in the CI, and doubt we will in future (if we do, it'll be via Guix, where this var would be redundant in any case). The CI environment doesn't (exactly) match the release build environment (and is semi-regularly changing), and the binaries produced in the CI don't match how we build release binaries, so there is no point trying to run these checks, especially as we add more involved tests, i.e #26953.

ACKs for top commit:
  josibake:
    code review ACK 6a936580d1
  TheCharlatan:
    ACK 6a936580d1

Tree-SHA512: c0eec61a4b873bac487ba9321b50116a215b4796bd7d416d98ffcd09969dbf635c2cb5aeb225c89d1e6462838fa2a48565048ebe730f48d76d3db46b64855a91
2023-05-22 09:52:27 +01:00
MarcoFalke fa1b3abc83
ci: Log qa-assets repo last commit
This documents the state in the CI output and may help debugging in case
of failure.
2023-05-22 10:02:40 +02:00
fanquake 5228223e1f
ci: remove MSAN getrandom syscall workaround
The corresponding workaround will also be dropped in oss-fuzz:
25946a5448/projects/bitcoin-core/build.sh (L49).
2023-05-20 17:20:06 +01:00
Hennadii Stepanov 98ea798411
ci, iwyu: Double maximum line length for includes 2023-05-20 13:16:26 +01:00
MarcoFalke fa953f15bf
build: Bump minimum supported GCC to g++-9
Also, update the code to use constexpr, which does not work in g++-8.

Also, drop the no longer needed build-aux/m4/l_filesystem.m4.
2023-05-18 12:24:40 +02:00
MarcoFalke fa69955e74
ci: Bump centos:stream8 to centos:stream9
This is required for the next commit. Also, drop CI_RETRY_EXE before
"dnf install", because it requires getopt, which will only be installed
later on via util-linux
2023-05-18 12:24:38 +02:00
MarcoFalke fa6a755d9f
ci: Document the false positive error for g++-9 2023-05-18 12:24:28 +02:00
fanquake 77b0a80ce7
Merge bitcoin/bitcoin#27682: build: Bump minimum supported Clang to clang-10
fa199ee614 ci: Drop NO_WERROR=1 for clang-10 build (MarcoFalke)
fad2c200f4 build: Bump minimum Clang to clang-10 (MarcoFalke)
fad7cfee8d doc: Remove outdated CentOS comment (MarcoFalke)

Pull request description:

  It doesn't make sense to support a minimum clang version that is difficult to install on all supported operating systems, which generally ship a later version:

  * Ubuntu Focal 20.04: https://packages.ubuntu.com/focal/clang-10 and https://packages.ubuntu.com/focal/clang-12
  * Debian Bullseye: https://packages.debian.org/bullseye/clang-13
  * CentOS 8 Stream: All Clang versions from 11.0 to 15.0

  Also, it allows to drop build code, which means it won't waste review when rolling over into cmake (`cmake/module/CheckStdFilesystem.cmake`).

ACKs for top commit:
  hebasto:
    ACK fa199ee614
  fanquake:
    ACK fa199ee614

Tree-SHA512: c1a0e8f191a6db866b8be3c9d254dc3f576fa021e2eaaeb68f3354554a8b38eaa90bbf9871ff92351b715e62a6b7b98cf94eba6dc53d7c951bddb6ad49ba7716
2023-05-18 11:05:08 +01:00
fanquake 4c3d67a2d0
Merge bitcoin/bitcoin#27571: ci: Run iwyu on all src files
ddddf4957b ci: Run iwyu on all src files (MarcoFalke)

Pull request description:

  This makes it easier to look at the CI output of a file without having to manually add it first to the list.

ACKs for top commit:
  hebasto:
    ACK ddddf4957b

Tree-SHA512: 342b52838ae45ea343731c30058cdd5595d5ea5601a1f396de4466ccdd63f7ab07b3a193df3669e4dca7cb535557dcc98f866b3cf986b98176b20ecead123868
2023-05-17 13:06:23 +01:00
fanquake 6a936580d1
ci: remove RUN_SECURITY_TESTS
We no-longer run any security/syymbol checks in the CI, and doubt we
will in future (if we do, it'll be via Guix, where this var would be
redundant in any case). The CI environment doesn't (exactly) match the
release build environment (and is semi-regularly changing), and the
binaries produced in the CI don't match how we build release binaries,
so there is no point trying to run these checks, especially as we add
more involved tests, i.e #26953.
2023-05-17 10:57:39 +01:00
MarcoFalke fa199ee614
ci: Drop NO_WERROR=1 for clang-10 build
This partially reverts commit 71383f2fad.
This should be fine, because if warnings are issues again in the future,
it can be disabled again, along with a list of the false warnings.
2023-05-17 10:55:03 +02:00
MarcoFalke fad2c200f4
build: Bump minimum Clang to clang-10 2023-05-17 10:30:41 +02:00
MarcoFalke fad7cfee8d
doc: Remove outdated CentOS comment
No longer applicable after CONFIG_SHELL must be explicitly set to dash
after commit fafc55a489.
2023-05-17 09:54:31 +02:00
fanquake 904631e0fc
Merge bitcoin/bitcoin#27667: ci: Remove unused errtrace trap ERR
fad09b703f ci: Remove unused errtrace trap ERR (MarcoFalke)

Pull request description:

  This was added in commit 069752b726, presumably at a time when the functional tests wouldn't capture stderr.

  Now that all tests capture and print stderr on failure, it can be removed. Reference:

  * Unit tests capture via `2>&1`:

  d7700d3a26/src/Makefile.test.include (L421)

  * Functional tests capture as well:

  d7700d3a26/test/functional/test_framework/test_node.py (L356)

ACKs for top commit:
  fanquake:
    ACK fad09b703f
  hebasto:
    ACK fad09b703f, tested on Ubuntu 22.04: I can still see warnings from the sanitizers in both unit and functional tests.

Tree-SHA512: 1e786eee432a7a50eb9f78b06b2b157321cc16f91b613e3b476e9e51572592fe4bcf4dc15df176e5f019f24497ac68cf332d2037b55b57498c93f4e19613163c
2023-05-16 15:28:04 +01:00
Hennadii Stepanov 5d49d98731
ci: Fix "Number of CPUs" output 2023-05-16 12:45:32 +01:00
MarcoFalke fad09b703f
ci: Remove unused errtrace trap ERR 2023-05-16 10:00:47 +02:00
fanquake 3a63ef5020
Merge bitcoin/bitcoin#27616: ci: Remove CI_EXEC bloat
fa01c3c59c ci: Remove CI_EXEC bloat (MarcoFalke)
fa8a428c92 move-only: Move almost all CI_EXEC code to 06_script_b.sh (MarcoFalke)

Pull request description:

  `CI_EXEC` has many issues:

  * It is roughly equivalent to `bash -c "$*"`, meaning that the full command will be treated as a single string, ignoring tokens.
  * It must be put in front of (almost) every command, making it easy to forget, hard to debug the resulting failure, and the code verbose.

  Fix all issues by removing it almost completely.

ACKs for top commit:
  TheCharlatan:
    ACK fa01c3c59c

Tree-SHA512: 4a65d61f5c35ca945d31f270dba3e96305fd83333a7713f0452c67f02a78e1901113e9f18d21e1dc016403c0033eb32038a9308d0a0ded7ee6b970d18381a1c2
2023-05-15 11:15:25 +01:00
MarcoFalke ddddf4957b
ci: Run iwyu on all src files
This makes it easier to look at the CI output of a file without having
to manually add it first.
2023-05-13 10:45:52 +02:00
TheCharlatan a498d699e3
refactor/iwyu: Complete includes for blockmanager_args 2023-05-10 19:07:30 +02:00
MarcoFalke fa01c3c59c
ci: Remove CI_EXEC bloat 2023-05-10 15:28:19 +02:00
MarcoFalke fa8a428c92
move-only: Move almost all CI_EXEC code to 06_script_b.sh
[WARN] The commit is obviously broken and will not run the CI system. In
the rare case this is hit in a git bisect, just skip the commit.

The goal here was to make it trivial to review with the git option:
--color-moved=dimmed-zebra

It is required to move everything into one file because "exit 0" will
otherwise stop working as intended when the containing bash script is no
longer executed with "source ...".

If there is desire to split up 06_script_b.sh into logical chunks in the
future, it will also be easier after the following commit.
2023-05-10 14:10:38 +02:00
MarcoFalke fa1dbd04ca
ci: Remove CI_EXEC bloat in test/06_script_b.sh 2023-05-05 08:51:43 +02:00
MarcoFalke fae8de926a
ci: Move CI container kill out of 06_script_b.sh
This cleans up 06_script_b.sh to only contain code to be executed inside
the CI pod, which avoids confusion and is needed for the next commit.
2023-05-05 08:50:39 +02:00
MarcoFalke fa7d75540e
ci: Pass full env to CI pod to avoid missing a var
Instead of enumerating each passed env var, just pass all. This avoids
the risk of missing to enumerate one. Also, it is less code.

The risk could be that an env var causes non-deterministic behavior, but
this can be fixed by explicitly excluding it once the issue is known.

Values with newlines can not be stored in the file and parsed by
docker/podman, so they are excluded.
2023-05-05 08:46:35 +02:00
fanquake f952e679cd
ci: remove usage of untrusted bpfcc-tools
We've migrated this job to Ubuntu 23.04, which
ships with newer versions of the tools:
https://packages.ubuntu.com/lunar/bpfcc-tools.
2023-05-02 12:02:45 +01:00
fanquake 1232c2f6b9
ci: use LLVM/clang-16 in native_asan job
Similar to #27298.
2023-05-02 12:02:23 +01:00
fanquake d89aca1bdb
Merge bitcoin/bitcoin#27483: Bump python minimum version to 3.8
fac395e5eb ci: Bump ci/lint/Dockerfile (MarcoFalke)
fa6eb65167 test: Use python3.8 pow() (MarcoFalke)
88881cf7ac Bump python minimum version to 3.8 (MarcoFalke)

Pull request description:

  There is no pressing reason to drop support for 3.7, however there are several maintenance issues:

  * There is no supported operating system that ships 3.7 by default. (debian:buster is EOL and unmaintained to the extent that it doesn't run in the CI environment. See https://github.com/bitcoin/bitcoin/pull/27340#issuecomment-1484988445)
  * Compiling python 3.7 from source is also unsupported on at least macos, according to https://github.com/bitcoin/bitcoin/pull/24017#issuecomment-1107820790
  * Recent versions of lief require 3.8, see https://github.com/bitcoin/bitcoin/pull/27507#issuecomment-1517561645

  Fix all maintenance issues by bumping the minimum.

ACKs for top commit:
  RandyMcMillan:
    ACK fac395e
  fjahr:
    ACK fac395e5eb
  fanquake:
    ACK fac395e5eb

Tree-SHA512: c198decdbbe29d186d73ea3f6549d8a38479383495d14a965a2f9211ce39637b43f13a4c2a5d3bf56e2d468be4bbe49b4ee8e8e19ec69936ff43ddf2b714c712
2023-04-28 10:22:20 +01:00
MarcoFalke fac395e5eb
ci: Bump ci/lint/Dockerfile
This bump should not be needed, see discussion starting at
https://github.com/bitcoin/bitcoin/pull/27483#issuecomment-1517739626
2023-04-21 14:31:13 +02:00
fanquake 669af32632
Merge bitcoin/bitcoin#27419: move-only: Extract common/args from util/system
be55f545d5 move-only: Extract common/args and common/config.cpp from util/system (TheCharlatan)

Pull request description:

  This pull request is part of the `libbitcoinkernel` project https://github.com/bitcoin/bitcoin/issues/24303 https://github.com/bitcoin/bitcoin/projects/18 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel". It is part of a series of patches splitting up the `util/system` files. Its preceding pull request is https://github.com/bitcoin/bitcoin/pull/27254.

  The pull request contains an extraction of ArgsManager related functions from util/system into their own common/ file.

  The background of this commit is an ongoing effort to decouple the libbitcoinkernel library from the ArgsManager. The ArgsManager belongs into the common library, since the kernel library should not depend on it. See [doc/design/libraries.md](https://github.com/bitcoin/bitcoin/blob/master/doc/design/libraries.md) for more information on this rationale.

ACKs for top commit:
  MarcoFalke:
    re-ACK be55f545d5  🚲
  ryanofsky:
    Code review ACK be55f545d5. Just small cleanups since the last review.
  hebasto:
    ACK be55f545d5, I have reviewed the code and it looks OK, I agree it can be merged.

Tree-SHA512: 90eb03334af0155b823030b4f2ecf286d35058d700ee2ddbbaa445be19e31eb0fe982656f35bd14ecee3ad2c3d0db3746855cb8f3777eff7253713e42873e111
2023-04-21 11:19:08 +01:00
MarcoFalke 88881cf7ac
Bump python minimum version to 3.8
Also, switch ci_native_qt5 to g++-9 (from g++-8) to work around bugs.
This should be fine, because the i686_centos task still checks for g++-8
compatibility.

See
https://github.com/bitcoin/bitcoin/pull/27483#issuecomment-1513477050
for the list of bugs.
2023-04-21 10:18:19 +02:00
fanquake 4de9c2a65f
ci: build libc++ with assertions in MSAN jobs 2023-04-19 11:54:44 +01:00
fanquake 23b8b2026a
ci: build libc++ in DEBUG mode in MSAN jobs 2023-04-19 11:54:44 +01:00
TheCharlatan be55f545d5
move-only: Extract common/args and common/config.cpp from util/system
This is an extraction of ArgsManager related functions from util/system
into their own common file.

Config file related functions are moved to common/config.cpp.

The background of this commit is an ongoing effort to decouple the
libbitcoinkernel library from the ArgsManager. The ArgsManager belongs
into the common library, since the kernel library should not depend on
it. See doc/design/libraries.md for more information on this rationale.
2023-04-19 10:48:30 +02:00
fanquake 5165984afc
Merge bitcoin/bitcoin#27340: ci: Use Cirrus CI dockerfile env
fa4a46de0b ci: Bump nowallet_libbitcoinkernel task to ubuntu:focal (MarcoFalke)
fabc7d90a9 ci: Use credits in more tasks (MarcoFalke)
facae3b149 ci: Use Cirrus CI dockerfile env (MarcoFalke)

Pull request description:

  Currently the CI env has many intermittent issues:

  * The Ubuntu package servers are frequently down
  * Occasionally other stuff is down, such as dnf, pip, or the android sdk
  * Installing packages is slower than downloading them, at least on Cirrus, which has a fast download speed

  Fix all issues by using the Cirrus CI dockerfile env.

ACKs for top commit:
  josibake:
    code review ACK fa4a46de0b

Tree-SHA512: fea5663f7b6dc1c4ea9f87188026ec542b9269bac8ee3398cd58d4df6c86a0af9d275f1876e03f92fb1f6166ec49b817d9e588e6fe1ed54b77592502c2eccd9d
2023-04-18 10:39:39 +01:00
fanquake 2c60826b50
ci: explicitly install libclang-rt-dev in valgrind jobs
This fixes some cases, i.e under --no-install-recommends, where
libclang-rt-dev wouldn't be installed, and configuring would then fail.

Followup to #27444.
2023-04-13 11:22:39 +01:00
fanquake ba29143d98
ci: use Debian Bookworm and Valgrind 3.19 in Valgrind jobs
https://packages.debian.org/bookworm/valgrind
2023-04-12 17:45:14 +01:00
MarcoFalke fa4a46de0b
ci: Bump nowallet_libbitcoinkernel task to ubuntu:focal
This is needed to work around
https://github.com/bitcoin/bitcoin/pull/27340#issuecomment-1484988445

The only change should be that python3.7 is bumped to 3.8, but this is
fine because ci/test/00_setup_env_native_qt5.sh still checks for
python3.7 compatibility.
2023-04-11 14:12:37 +02:00
fanquake 676671527f
test: LLVM/Clang 16 for MSAN jobs
Sync up with other CI infra.
2023-04-10 10:57:05 +01:00
MarcoFalke fa5af94de6
ci: Run base install at most once
This should avoid errors when running it twice. For example, network
errors on the second invocation of 'apt update'; or unguarded
modifications such as APPEND_APT_SOURCES_LIST, which will append the
same string repeatedly.

The base install may be run twice in Cirrus CI with dockerfiles, or
locally when running twice with DANGER_RUN_CI_ON_HOST specified.
2023-04-06 11:52:30 +02:00
fanquake 5a8bd45056
Merge bitcoin/bitcoin#27423: ci: fix git dubious permissions error
ed4a8339b8 ci: fix git dubious permissions error (josibake)

Pull request description:

  fixes https://github.com/bitcoin/bitcoin/pull/27376#issuecomment-1496449588

  this appears to be caused by a more recent version of git being sensitive to mismatched permissions on directories. we didn't notice this before because we were using two separate user accounts to fix up dir permissions in the container , but the second account was removed in #27376

  there might be a more elegant way to do this, but this does the trick and seems to be the way others are fixing this issue around the internets.

ACKs for top commit:
  RandyMcMillan:
    ACK ed4a833
  hebasto:
    re-ACK ed4a8339b8

Tree-SHA512: dad467deca101a24f3ed34b3e26a1db5099a5bd5c3e9c9a22771c59848f7d7e7843c7386348e6fdf86d5a556e4706e5e20005d7a6637193e1c8aef7a5ff7fb19
2023-04-06 10:04:01 +01:00
josibake ed4a8339b8
ci: fix git dubious permissions error 2023-04-05 14:25:13 +02:00
fanquake a56c96507a
ci: use clang-16 in tidy task 2023-04-05 11:43:42 +01:00
MarcoFalke fafe3a8e38
ci: Remove second user account 2023-03-31 08:28:38 +02:00
fanquake 45be6af694
Merge bitcoin/bitcoin#27333: ci: cleanup of CI_EXEC & CI_EXEC_ROOT
b5ef1419ec ci: cleanup of CI_EXEC & CI_EXEC_ROOT (refs #27321) (Vasil Stoyanov)

Pull request description:

  Basically it removes the above-mentioned env-vars as per MarcoFalke's instructions. The only deviation from the plan laid out there was that I double-quoted the last instance of $ANDROID_HOME for the sake of consistency and future-proofing and the rest of the non-quoted vars due to lint failing the build.

  Fixes #27321.

ACKs for top commit:
  josibake:
    ACK b5ef1419ec
  hernanmarino:
    untested ACK b5ef1419ec. LGTM

Tree-SHA512: a79776bf64a2fa8b38195cc84445e171fd689f156aac5a1e5d39040300567eb9f4c2ebd00fbf3fa0e55b68793f8f752d94f7d817f6097ed9dd3a8ea57651b981
2023-03-30 18:55:27 +01:00
fanquake a634c288c3
ci: use LLVM/clang-16 in native_fuzz (ASAN) job
Similar to #27298.
2023-03-29 16:05:59 +01:00
fanquake 220008604f
Merge bitcoin/bitcoin#27298: ci: Use TSan new runtime (llvm-16, take 3)
faf4aca15a ci: Use TSan new runtime (llvm-16, take 3) (MarcoFalke)

Pull request description:

  The previous two attempts failed:
  * llvm-14: Failed in https://github.com/bitcoin/bitcoin/pull/24572
  * llvm-15: Failed in https://github.com/bitcoin/bitcoin/pull/26775

  However, now that the bug is known and fixed, it should be good to go. See also https://github.com/bitcoin/bitcoin/pull/26775#issuecomment-1380590669

ACKs for top commit:
  fanquake:
    ACK faf4aca15a - I still see [this](https://github.com/bitcoin/bitcoin/pull/27298#issuecomment-1480041089) failure on aarch64, but that isn't really a regression, as running this tests was already broken for me. I'll open a separate issue, and we can follow up.

Tree-SHA512: 372b53c4d42ca7f527dae4a2b5bc5ab33c816930daf7a3479d20ea7749159a0b19cfd8d76244b95b03130e4a3d12ddbbb74668b8f7e9fc272cf1084f53b7ff9b
2023-03-28 14:17:30 +01:00
MarcoFalke faf4aca15a
ci: Use TSan new runtime (llvm-16, take 3) 2023-03-28 10:01:47 +02:00
Vasil Stoyanov b5ef1419ec ci: cleanup of CI_EXEC & CI_EXEC_ROOT (refs #27321) 2023-03-27 20:45:17 +03:00
MarcoFalke faf8dc496e
fuzz: Remove legacy int parse fuzz tests 2023-03-27 16:37:31 +02:00
MarcoFalke faa0839837
ci: Cache more stuff in the ci images: msan, iwyu, pip, sdks 2023-03-24 10:19:25 +01:00
Hennadii Stepanov 8fe27fbed8
ci: Use clang-15 in "tidy" task 2023-03-22 15:21:39 +00:00
Ben Woosley e7333b420e
refactor: Extract util/exception from util/system
This is a minimal extraction of a single function, but also the only use
of std::exception in util/system.

The background of this commit is an ongoing effort to decouple the
libbitcoinkernel library from the ArgsManager defined in system.h.
Moving the function out of system.h allows including it from a separate
source file without including the ArgsManager definitions from system.h.
2023-03-13 17:09:47 +01:00
Andrew Chow 23e2bfcbc4
Merge bitcoin/bitcoin#25696: build: Re-enable external signer on Windows
1a0d8e178c build: Re-enable external signer on Windows (Hennadii Stepanov)
989451d068 configure: Detect compatibility of Boost.Process rather than hardcode non-Windows (Luke Dashjr)

Pull request description:

  As https://github.com/boostorg/process/issues/207 has been resolved, it is possible now to re-enable external signer on Windows when cross-compiling.

  Guix build hashes:
  ```
  78f69ea7e0dbc8338981a92c0352220ccd7c2272d8cbff6a3b082a1412a935c5  guix-build-1a0d8e178c7b/output/aarch64-linux-gnu/SHA256SUMS.part
  ee17456ec818ddf5a175182508966e622573ccb518807cca43a40fa1dceda092  guix-build-1a0d8e178c7b/output/aarch64-linux-gnu/bitcoin-1a0d8e178c7b-aarch64-linux-gnu-debug.tar.gz
  5080551bde379c746cc67b10429aef33b9f9e49d2d4e21ee1c3bfd9c1c845d46  guix-build-1a0d8e178c7b/output/aarch64-linux-gnu/bitcoin-1a0d8e178c7b-aarch64-linux-gnu.tar.gz
  dfab220ce76a40bf7dcf07aab352a616a91b516503639455fe7e1b137bad3e85  guix-build-1a0d8e178c7b/output/arm-linux-gnueabihf/SHA256SUMS.part
  516ceb822571a8bd88fe107dca434ef596b1e4328ccbda1d51e1d482d3050396  guix-build-1a0d8e178c7b/output/arm-linux-gnueabihf/bitcoin-1a0d8e178c7b-arm-linux-gnueabihf-debug.tar.gz
  21325380638f817107c203b9a1aedb808d1a4a2b4041493753ca4cbf19aa4f2c  guix-build-1a0d8e178c7b/output/arm-linux-gnueabihf/bitcoin-1a0d8e178c7b-arm-linux-gnueabihf.tar.gz
  cf48ed78fcfceaeb3610ccf22326d735a129dcbf9d50b557b3de359169aefdfd  guix-build-1a0d8e178c7b/output/arm64-apple-darwin/SHA256SUMS.part
  d4d51e136148bac6a20bb3adb402c499967647736acb420bfdeb71603aba57da  guix-build-1a0d8e178c7b/output/arm64-apple-darwin/bitcoin-1a0d8e178c7b-arm64-apple-darwin-unsigned.dmg
  95bb62d24f860e08a392ddb74d5860ccf27e8baa183e6749af877d26a3bd6b0b  guix-build-1a0d8e178c7b/output/arm64-apple-darwin/bitcoin-1a0d8e178c7b-arm64-apple-darwin-unsigned.tar.gz
  68da4c92f37bb802df37141af194f47c16da1d84f77a0fbb1016013ae0338502  guix-build-1a0d8e178c7b/output/arm64-apple-darwin/bitcoin-1a0d8e178c7b-arm64-apple-darwin.tar.gz
  6704e38c2d3f11321403797598d05f062648fec6f2d76900ba250dab481e29da  guix-build-1a0d8e178c7b/output/dist-archive/bitcoin-1a0d8e178c7b.tar.gz
  64b936bc90d1e01fe8f276511edc9bb945dcebe70332aa37d3a786348443b8e7  guix-build-1a0d8e178c7b/output/powerpc64-linux-gnu/SHA256SUMS.part
  3d03532e54b6e42498ea240c86b8567e94fd462f56087b869c3d6f09e2dde878  guix-build-1a0d8e178c7b/output/powerpc64-linux-gnu/bitcoin-1a0d8e178c7b-powerpc64-linux-gnu-debug.tar.gz
  c5843d79a58b0a864fe723458dab4eee54ad11f4b1f7960975b086eeedc0d541  guix-build-1a0d8e178c7b/output/powerpc64-linux-gnu/bitcoin-1a0d8e178c7b-powerpc64-linux-gnu.tar.gz
  f861ff519bd5e3d6d5ce1646ee0a06bcef1288ddb804a4a600e4dbfe5d5be521  guix-build-1a0d8e178c7b/output/powerpc64le-linux-gnu/SHA256SUMS.part
  5f477da21980dbcf9696081903dc1ba8a3f79ce3579641d208e69a6f598c8eb9  guix-build-1a0d8e178c7b/output/powerpc64le-linux-gnu/bitcoin-1a0d8e178c7b-powerpc64le-linux-gnu-debug.tar.gz
  b3757b11c614136934158acea5139e8abd0c5c9cdfda72ae44db436f21716b33  guix-build-1a0d8e178c7b/output/powerpc64le-linux-gnu/bitcoin-1a0d8e178c7b-powerpc64le-linux-gnu.tar.gz
  1c21bdb17fe3436e685e88c62423e630fe2b3c41dd00025a99fd80d97817ac2f  guix-build-1a0d8e178c7b/output/riscv64-linux-gnu/SHA256SUMS.part
  f36ae98473f086ae8f0dc66223b5ec407d57dc4d8d45ae284401520ff5c0b273  guix-build-1a0d8e178c7b/output/riscv64-linux-gnu/bitcoin-1a0d8e178c7b-riscv64-linux-gnu-debug.tar.gz
  1603e4d0e869eb47a1dc2d26b67772d0016d90f7ba5e50d2009365cc02cb8169  guix-build-1a0d8e178c7b/output/riscv64-linux-gnu/bitcoin-1a0d8e178c7b-riscv64-linux-gnu.tar.gz
  f86ef652102f022827b70477bffa0a44008c6300cf62ca7b3595146cf2ed91ba  guix-build-1a0d8e178c7b/output/x86_64-apple-darwin/SHA256SUMS.part
  f84d435d8e4709bf29bc7ac7ed8dc6b8af4077cef05e520b468b2896ce10876a  guix-build-1a0d8e178c7b/output/x86_64-apple-darwin/bitcoin-1a0d8e178c7b-x86_64-apple-darwin-unsigned.dmg
  af2aab969b7ed7aeea0e02adbcc9e3b438086bf76b6bfc36146c53e05a27bd57  guix-build-1a0d8e178c7b/output/x86_64-apple-darwin/bitcoin-1a0d8e178c7b-x86_64-apple-darwin-unsigned.tar.gz
  32a5109ba28ab74ff66238e6a8f8a04e455ebce382a3be287df92a227818fe72  guix-build-1a0d8e178c7b/output/x86_64-apple-darwin/bitcoin-1a0d8e178c7b-x86_64-apple-darwin.tar.gz
  377462e9a96f4aba72c915dd5df5159a4301a1fa8ed0ee48faa6c71573de80c3  guix-build-1a0d8e178c7b/output/x86_64-linux-gnu/SHA256SUMS.part
  a3bf62e828d2350a483b2d16205014f66e8884597b0b72e178042a958c548336  guix-build-1a0d8e178c7b/output/x86_64-linux-gnu/bitcoin-1a0d8e178c7b-x86_64-linux-gnu-debug.tar.gz
  66cda980188ea1941a7d66c8b03c447580af33db55abe3bbe3581823ae0534a3  guix-build-1a0d8e178c7b/output/x86_64-linux-gnu/bitcoin-1a0d8e178c7b-x86_64-linux-gnu.tar.gz
  2117f0dd9baeb4d585f841592e94c088f4487bf2008b8f281d0c3ceee92ff6cc  guix-build-1a0d8e178c7b/output/x86_64-w64-mingw32/SHA256SUMS.part
  d40d5dec3287f467c42232c05d82f7fb538cda34bd2e63ff7e1876f471c3a790  guix-build-1a0d8e178c7b/output/x86_64-w64-mingw32/bitcoin-1a0d8e178c7b-win64-debug.zip
  92dcc92765fbc07b1cc8258bfa69280541e1b4553cc41fed18672c2c6931d5c0  guix-build-1a0d8e178c7b/output/x86_64-w64-mingw32/bitcoin-1a0d8e178c7b-win64-setup-unsigned.exe
  a6dd9b4d29f21d3a18cf64556cb03446ef17bf801eb6ac257b65d27cbd95080f  guix-build-1a0d8e178c7b/output/x86_64-w64-mingw32/bitcoin-1a0d8e178c7b-win64-unsigned.tar.gz
  a4022e595d955198f73530473ef8e90a708746089ee2dd27de794176873330c1  guix-build-1a0d8e178c7b/output/x86_64-w64-mingw32/bitcoin-1a0d8e178c7b-win64.zip
  ```

ACKs for top commit:
  Sjors:
    tACK 1a0d8e178c
  achow101:
    ACK 1a0d8e178c

Tree-SHA512: db7319259b1e1571cfab4bb3b99ae10a2f744e62757cae5059fd6f4dd6d5586eb09feb63a0c4bb07f7128b283f1dc281ed435224bc8e40da577fd4f04cde489a
2023-03-08 21:01:53 -05:00
fanquake 2de0559f2c
Merge bitcoin/bitcoin#27189: util: Use steady clock in SeedStrengthen, FindBestImplementation, FlushStateToDisk
fa1b4e5c32 Use steady clock in FlushStateToDisk (MarcoFalke)
1111e2f8b4 Use steady clock in SeedStrengthen and FindBestImplementation (MarcoFalke)

Pull request description:

  There may be a theoretical deadlock for the duration of the offset when the system clock is adjusted into a past time while executing `SeedStrengthen`.

  Fix this by using steady clock.

  Do the same in `FindBestImplementation`, which shouldn't be affected, because it discards outlier measurements. However, doing the same there for consistency seems fine.

  Do the same in `FlushStateToDisk`, which should make the flushes more steady, if the system clock is adjusted by a large offset.

ACKs for top commit:
  john-moffett:
    ACK fa1b4e5c32
  willcl-ark:
    ACK fa1b4e5c3

Tree-SHA512: cc625e796b186accd53222bd64eb57d0512bc7e588312d254349b542bbc5e5daac348ff2b3b3f7dc5ae0bbbae2ec11fdbf3022cf2164211633765a4b0108e83e
2023-03-08 08:48:41 +01:00
Andrew Chow fc037c8c83
Merge bitcoin/bitcoin#27150: Deduplicate bitcoind and bitcoin-qt init code
802cc1ef53 Deduplicate bitcoind and bitcoin-qt init code (Ryan Ofsky)
d172b5c671 Add InitError(error, details) overload (Ryan Ofsky)
3db2874bd7 Extend bilingual_str support for tinyformat (Ryan Ofsky)
c361df90b9 scripted-diff: Remove double newlines after some init errors (Ryan Ofsky)

Pull request description:

  Add common InitConfig function to deduplicate bitcoind and bitcoin-qt code reading config files and creating the datadir.

  Noticed the duplicate code while reviewing #27073 and want to remove it because difference in bitcoin-qt and bitcoind behavior make it hard to evaluate changes like #27073

  There are a few minor changes in behavior:

  - In bitcoin-qt, when there is a problem reading the configuration file, the GUI error text has changed from "Error: Cannot parse configuration file:" to "Error reading configuration file:" to be consistent with bitcoind.
  - In bitcoind, when there is a problem reading the settings.json file, the error text has changed from "Failed loading settings file" to "Settings file could not be read" to be consistent with bitcoin-qt.
  - In bitcoind, when there is a problem writing the settings.json file, the error text has changed from "Failed saving settings file" to "Settings file could not be written" to be consistent with bitcoin-qt.
  - In bitcoin-qt, if there datadir is not accessible (e.g. no permission to read), there is an normal error dialog showing "Error: filesystem error: status: Permission denied [.../settings.json]", instead of an uncaught exception.

ACKs for top commit:
  Sjors:
    Light review ACK 802cc1ef53
  TheCharlatan:
    ACK 802cc1ef53
  achow101:
    ACK 802cc1ef53

Tree-SHA512: 9c78d277e9ed595fa8ce286b97d2806e1ec06ddbbe7bd3434bd9dd7b456faf8d989f71231e97311f36edb9caaec645a50c730bd7514b8e0fe6e6f7741b13d981
2023-03-07 13:05:01 -05:00
glozow 2a0c05defd
Merge bitcoin/bitcoin#27209: ci: Remove unused EXPECTED_TESTS_DURATION_IN_SECONDS env var
3fffff50f6 ci: Remove unused EXPECTED_TESTS_DURATION_IN_SECONDS env var (MarcoFalke)

Pull request description:

  Remove long unused travis leftover

ACKs for top commit:
  fanquake:
    ACK 3fffff50f6
  dergoegge:
    ACK 3fffff50f6

Tree-SHA512: 34b85a18c0d34f54bbc6ff5c2454307318e4747145e11f52f08baddefef9e1b80d8f6c85efcad97e575380162fd193f2aa837e99b156ed7e3e95370b635ddf4e
2023-03-06 14:00:36 +00:00
MarcoFalke 1111e2f8b4
Use steady clock in SeedStrengthen and FindBestImplementation 2023-03-02 14:48:28 +01:00
Ryan Ofsky 802cc1ef53 Deduplicate bitcoind and bitcoin-qt init code
Add common InitConfig function to deduplicate bitcoind and bitcoin-qt code
reading config files and creating the datadir.

There are a few minor changes in behavior:

- In bitcoin-qt, when there is a problem reading the configuration file, the
  GUI error text has changed from "Error: Cannot parse configuration file:" to
  "Error reading configuration file:" to be consistent with bitcoind.
- In bitcoind, when there is a problem reading the settings.json file, the
  error text has changed from "Failed loading settings file" to "Settings
  file could not be read" to be consistent with bitcoin-qt.
- In bitcoind, when there is a problem writing the settings.json file, the
  error text has changed from "Failed saving settings file" to "Settings file
  could not be written" to be consistent with bitcoin-qt.
- In bitcoin-qt, if there datadir is not accessible (e.g. no permission to read),
  there is an normal error dialog showing "Error: filesystem error: status:
  Permission denied [.../settings.json]", instead of an uncaught exception
2023-02-28 12:04:47 -05:00
Andrew Chow a245429d68
Merge bitcoin/bitcoin#26940: test: create random and coins utils, add amount helper, dedupe add_coin
4275195606 De-duplicate add_coin methods to a test util helper (Jon Atack)
9d92c3d7f4 Create InsecureRandMoneyAmount() test util helper (Jon Atack)
81f5ade2a3 Move random test util code from setup_common to random (Jon Atack)

Pull request description:

  - Move random test utilities from `setup_common` to a new `random` file, as many tests don't use this code.

  - Create a helper to generate semi-random CAmounts up to `MONEY_RANGE` rather than only uint32, and use the helper in the unit tests.

  - De-duplicate a shared `add_coin` method by extracting it to a `coins` test utility.

ACKs for top commit:
  pinheadmz:
    ACK 4275195606
  achow101:
    ACK 4275195606
  john-moffett:
    ACK 4275195606

Tree-SHA512: 3ed974251149c7417f935ef2f8865aa0dcc33b281b47522b0f96f1979dff94bb8527957f098fe4d210f40d715c00f29512f2ffe189097102229023b7284a3a27
2023-02-17 17:28:14 -05:00
merge-script 141115a060
Merge bitcoin/bitcoin#27033: ci: Cache stuff in volumes, not host folders
fa8e92c022 doc: Update ci docs (721217.xyz)
5fffff54e9 ci: Cache stuff in volumes, not host folders (MarcoFalke)

Pull request description:

  Storing cached stuff in host system folders may lead to unexpected issues when the ci-built stuff is used for a non-ci build or a ci task leaks into another ci task.

ACKs for top commit:
  john-moffett:
    ACK fa8e92c022

Tree-SHA512: 8b0c9019452fbe507a272c1037c3dce3c178c21f85ab1096ed3372ad9d4b3c7aa27d89e5bf80c9a6260ea652e0268be0cbe61d6a4fcb3add569fa38076d32287
2023-02-13 11:53:50 +01:00
721217.xyz fa8e92c022
doc: Update ci docs 2023-02-10 17:05:39 +01:00
Jon Atack 4275195606 De-duplicate add_coin methods to a test util helper 2023-02-09 15:03:36 -08:00
Hennadii Stepanov 887bb53b67
ci: Use the latest Ubuntu LTS for "ARM64 Android APK" task 2023-02-08 14:15:13 +00:00
MarcoFalke 5fffff54e9
ci: Cache stuff in volumes, not host folders 2023-02-08 11:21:23 +01:00
Andrew Chow 52ddbd52f9
Merge bitcoin/bitcoin#26345: refactor: modernize the implementation of uint256.*
935acdcc79 refactor: modernize the implementation of uint256.* (pasta)

Pull request description:

  - Constructors of uint256 to utilize Span instead of requiring a std::vector
  - converts m_data into a std::array
  - Prefers using `WIDTH` instead of `sizeof(m_data)`
  - make all the things constexpr
  - replace C style functions with c++ equivalents
      - memset -> std::fill
          This may also be replaced by std::memset, but I think that std::fill is more idiomatic of modern c++ and readable.
      - memcpy -> std::copy
          Note: In practice, implementations of std::copy avoid multiple assignments and use bulk copy functions such as std::memmove if the value type is TriviallyCopyable and the iterator types satisfy LegacyContiguousIterator. (https://en.cppreference.com/w/cpp/algorithm/copy)
          This could also likely be replaced by std::memcpy, but as said above, I believe the using std::copy is the more c++ way to do anything and is almost guaranteed to compile to the same asm
      - memcmp -> std::memcmp

ACKs for top commit:
  achow101:
    ACK 935acdcc79
  hebasto:
    Approach ACK 935acdcc79.
  aureleoules:
    reACK 935acdcc79
  john-moffett:
    ACK 935acdcc79
  stickies-v:
    Approach ACK 935acdcc7

Tree-SHA512: 4f1ba54ff2198eea0e505d41e73d552c84c60f6878d5c85a94a8ab57f39afc94ef8d79258e7afd01fa84ec2a99f4404bb877eecd671f65e1ee9273f3129fc650
2023-02-06 13:56:51 -05:00
fanquake c2028f98ae
Merge bitcoin/bitcoin#27012: ci: Print iwyu patch in git diff format
fa6986a66b ci: Print iwyu patch in git diff format (MarcoFalke)

Pull request description:

  Seems more dev friendly to also have a patch to copy-paste

ACKs for top commit:
  hebasto:
    ACK fa6986a66b, tested on Ubuntu 22.04 locally.
  fanquake:
    ACK fa6986a66b - did not test but example CI output looks ok.
  stickies-v:
    utACK fa6986a66b

Tree-SHA512: 7cfd8584bf12e03c28af23f4712c6bcafd648d87ddb92788b9cd35455b2db49f4bd4aef8ad4711f75c7f11ad2bb2492c2eb6044007086c20e36016575c060603
2023-02-02 15:48:20 +00:00
MarcoFalke 3fffff50f6
ci: Remove unused EXPECTED_TESTS_DURATION_IN_SECONDS env var 2023-02-02 16:26:16 +01:00
MarcoFalke b3ef329199
Merge bitcoin/bitcoin#26976: ci: Cache package manager install step
fa486de212 ci: Cache package manager install step (MarcoFalke)

Pull request description:

  Use the local podman or docker image cache to skip the slow `apt` step

ACKs for top commit:
  jamesob:
    ACK fa486de212 ([`jamesob/ackr/26976.1.MarcoFalke.ci_cache_package_manager`](https://github.com/jamesob/bitcoin/tree/ackr/26976.1.MarcoFalke.ci_cache_package_manager))

Tree-SHA512: 3495346c6c862b63296d2691cc492bf52a0a99ee7fae798887c792609904546013eba788045cd508a5f669f2c52e3479c122c18a5275c87af38237a1b5c9da17
2023-02-02 16:09:23 +01:00
fanquake 71383f2fad
ci: avoid using -Werror for older compilers
Don't enable `-Werror` (in the CI) for compilers at least older than
our current release compiler (GCC 10). It provides little-to-no value,
other than turning compiler bugs & false positives into build failures,
and we aren't going to mutate perfectly fine/working code, for the sake
of avoid a warning that shouldn't even exist.

I also do not see the point of playing whack-a-mole and turning off various
warnings/trying to further work around the broken compiler, just to
acheive warningless builds for the sake of warningless builds.

One anecdote from "How SQLite Is Tested":
> Static analysis has found a few bugs in SQLite, but those are the
> exceptions. More bugs have been introduced into SQLite while trying
> to get it to compile without warnings than have been found by static
> analysis.

https://www.sqlite.org/testing.html.
2023-02-01 15:22:24 +00:00
MarcoFalke fa6986a66b
ci: Print iwyu patch in git diff format 2023-02-01 14:11:48 +01:00
fanquake b5868f4b1f
Merge bitcoin/bitcoin#23670: build: Build minisketch test in `make check`, not in `make`
6d58117a31 build: Build minisketch test in `make check`, not in `make` (Hennadii Stepanov)

Pull request description:

  On master (d1e42659bb):
  ```
  $ ./autogen.sh && ./configure --without-gui --disable-wallet && make clean
  $ make 2>&1 | grep LD | grep -v .la
    CXXLD    bitcoind
    CXXLD    bitcoin-cli
    CXXLD    bitcoin-tx
    CXXLD    bitcoin-util
    CXXLD    test/test_bitcoin
    CXXLD    bench/bench_bitcoin
    CXXLD    minisketch/test
    CXXLD    test/fuzz/fuzz
    CXXLD    univalue/test/object
    CXXLD    univalue/test/unitester
  $ make check 2>&1 | grep LD
    CCLD     exhaustive_tests
    CCLD     tests
  ```

  With this PR:
  ```
  $ ./autogen.sh && ./configure --without-gui --disable-wallet && make clean
  $ make 2>&1 | grep LD | grep -v .la
    CXXLD    bitcoind
    CXXLD    bitcoin-cli
    CXXLD    bitcoin-tx
    CXXLD    bitcoin-util
    CXXLD    test/test_bitcoin
    CXXLD    bench/bench_bitcoin
    CXXLD    test/fuzz/fuzz
    CXXLD    univalue/test/object
    CXXLD    univalue/test/unitester
  $ make check 2>&1 | grep LD
    CXXLD    minisketch/test
    CCLD     exhaustive_tests
    CCLD     tests
  ```

  In fact, this PR restores behavior that was before bitcoin/bitcoin#22646, and that behavior looks more optimal.

  As an outcome, the `contrib/guix/libexec/build.sh` does not spend resources to build binaries which are not a part of the release package.

ACKs for top commit:
  TheCharlatan:
    ACK 6d58117a31

Tree-SHA512: 4957c8f88a01aca005813bf4c1c26f433756bf68ea0c958481c638ead229fa8e23ecae3a8ac31ea555876ba6f2cc10ecd91caf2e2f664de5cb529ec05fb38fa7
2023-01-31 17:55:44 +00:00
fanquake 7241b936c5
Merge bitcoin/bitcoin#26965: refactor: Remove stray cs_main redundant declaration
faba08b5b4 refactor: Remove stray cs_main redundant declaration (MarcoFalke)
fa02591edf doc: Export threadsafety.h from sync.h (MarcoFalke)

Pull request description:

  Looks like this was forgotten when introducing kernel/cs_main ?

  Also, there is a commit to export threadsafety.h from sync.h.

ACKs for top commit:
  hebasto:
    ACK faba08b5b4

Tree-SHA512: 0aa58e7693b6fcd504f9da7339f8baa463a6407f67b27f68002db705f4642321ac3765f16c3d906c925ee24085591b79160a62fa5f4aaf6f2e5dcc788411800d
2023-01-30 17:27:44 +00:00
fanquake 228edafc66
Merge bitcoin/bitcoin#23619: build: Propagate user-defined flags to host packages
a3a2bd9e8a ci: Drop no longer needed package-specific flags (Hennadii Stepanov)
071eef1e97 build: Propagate user-defined flags to host packages (Hennadii Stepanov)

Pull request description:

  On master (4f8b1f8759) `{CPP,C,CXX,LD}FLAGS` that are specified in the command line are not propagated to packages:
  ```
  $ make --no-print-directory -C depends print-libevent_cxxflags CXXFLAGS=-some-fancy-flag
  libevent_cxxflags=-pipe -O2
  ```

  This PR:
  - propagates `{CPP,C,CXX,LD}FLAGS` to host packages:
  ```
  $ make --no-print-directory -C depends print-libevent_cxxflags CXXFLAGS=-some-fancy-flag
  libevent_cxxflags= -some-fancy-flag
  ```
  - does not propagate `{CPP,C,CXX,LD}FLAGS` to native packages:
  ```
  $ make --no-print-directory -C depends print-native_b2_cxxflags CXXFLAGS=-some-fancy-flag
  native_b2_cxxflags=
  ```
  - actually addresses the https://github.com/bitcoin/bitcoin/pull/23551#issuecomment-973896518

ACKs for top commit:
  TheCharlatan:
    Code review ACK a3a2bd9e8a

Tree-SHA512: 243d6b1b0e9c5de46debc36de62a77b6b4d6f638940fd530040c219956ec624e321b0c25290fed164e3a8c88befa7b97b20f765d7b9a428c269b3720f21da099
2023-01-30 14:32:32 +00:00
MarcoFalke fa486de212
ci: Cache package manager install step 2023-01-28 17:59:07 +01:00
MarcoFalke faba08b5b4
refactor: Remove stray cs_main redundant declaration 2023-01-25 09:56:26 +01:00
MarcoFalke fa88d42123
ci: Fix APPEND_APT_SOURCES_LIST trying to modify the host system 2023-01-24 20:33:55 +01:00
MarcoFalke 250598a905
Merge bitcoin/bitcoin#26906: test: add an easy way to run linters locally
b68e5a7fef lint: specify the right commit range when running locally (James O'Beirne)
dff7ed5732 test: add an easy way to run linters locally (James O'Beirne)

Pull request description:

  Adds a Dockerfile configuration ~~(originally written mostly by fanquake)~~ that allows straightforward running of linters with compatible versions locally. This removes a ton of annoyance when trying to appease CI, because many of the linter versions are quite old and difficult to maintain locally.

  I realize that people may not be thrilled to add more ancillary tooling to the repo, but I think this makes a lot of sense given the linter versions listed in this container configuration are dictated by this repo (within the CI configuration), so having these things live in two separate places is a recipe for version mismatches.

  Eventually we can likely just use this container on CI directly to avoid any chance of inconsistencies between local dev experience and CI.

ACKs for top commit:
  aureleoules:
    ACK b68e5a7fef
  stickies-v:
    ACK b68e5a7fe
  john-moffett:
    ACK b68e5a7fef

Tree-SHA512: 7ef7a5dae023d81fdb6296d5d92dfa074ee321c7993e607c9f014d0f21c91558611aa00fc3ce1edc7b5e68371aea0d27fa1931291a79bb867a6c783bb536775f
2023-01-19 19:01:28 +01:00
James O'Beirne b68e5a7fef lint: specify the right commit range when running locally
When running lints on Cirrus, a special envvar is set ($CIRRUS_PR);
emulate this when running linters locally by setting $LOCAL_BRANCH
to any value.
2023-01-18 09:48:14 -05:00
James O'Beirne dff7ed5732 test: add an easy way to run linters locally
Adds a Dockerfile configuration
that allows straightforward running of linters with compatible versions
locally. This removes a ton of annoyance when trying to appease CI,
because many of the linter versions are quite old and difficult to
maintain locally.

I realize that people may not be thrilled to more ancillary tooling to
the repo, but I think this makes a lot of sense given the linter
versions listed in this container configuration are dictated by this
repo (within the CI configuration), so having these things live in
two separate places is a recipe for version mismatches.

Eventually we can likely just use this container on CI directly to avoid
any chance of inconsistencies between local dev experience and CI.
2023-01-18 09:48:08 -05:00
MarcoFalke dddd462137
Bump minimum python version to 3.7 2023-01-18 12:59:11 +01:00
fanquake c9db788a11
Merge bitcoin/bitcoin#26908: ci: Add missing lint dependency (gpg)
fa87d71872 ci: Add missing lint dependency (MarcoFalke)

Pull request description:

  Also, document each dependency.

  Adding `gpg` avoids errors when running a release or dev branch in the CI:

  01ec5308bf/ci/lint/06_script.sh (L30-L42)

  ```
  bash: line 1: gpg: command not found
  ```

  https://cirrus-ci.com/task/4582854860996608?logs=lint#L185

ACKs for top commit:
  hebasto:
    ACK fa87d71872

Tree-SHA512: 869a3d2feab764b2c8d47d481359680a1d2c54a33b13ca26c5f8ce56cf2f368d4c74637dcbc53fdbf323f10940965c1c0e592e2fb4ce725d5cd467e77e62b6e5
2023-01-18 11:41:12 +00:00
MarcoFalke faddb7373a
ci: Bump --combinedlogslen to debug intermittent issues 2023-01-18 09:35:47 +01:00
MarcoFalke fa87d71872
ci: Add missing lint dependency 2023-01-17 19:09:44 +01:00
MarcoFalke 01ec5308bf
Merge bitcoin/bitcoin#26716: ci: Build python from source in "lint" task
123043e99c ci: Bump lint task image to Ubuntu Jammy (Hennadii Stepanov)
9b86114058 ci: Use pyenv's `python-build` to install Python in lint task (Hennadii Stepanov)

Pull request description:

  This PR:
  - is an alternative to bitcoin/bitcoin#26581 and bitcoin/bitcoin#26637
  - closes bitcoin/bitcoin#26548

  Key advantages of this PR over others:
  - it uses pyenv's `python-build` [standalone](https://github.com/pyenv/pyenv/tree/master/plugins/python-build#using-python-build-standalone)
  - requires no additional computational resources

  Note for testing. The lint task must success regardless of whether the `python_cache` is populated or invalidated.

ACKs for top commit:
  MarcoFalke:
    ACK 123043e99c
  fanquake:
    ACK 123043e99c

Tree-SHA512: ba0fcdd4f2939a59692b173dcd1f5704444cfcfbb8111538c6f8160056d0536bba250e4f9b0f8c66f8b454e52034bf36ffe6afae76cdc0f7cc5b58b576d790ba
2023-01-17 18:35:47 +01:00
Hennadii Stepanov 123043e99c
ci: Bump lint task image to Ubuntu Jammy 2023-01-16 13:30:00 +00:00
Hennadii Stepanov 9b86114058
ci: Use pyenv's `python-build` to install Python in lint task 2023-01-16 13:29:44 +00:00
MarcoFalke fa0584eb82
ci: Stop and remove CI container 2023-01-11 10:49:39 +01:00
MarcoFalke fa5dccba32
scripted-diff: ci: Rework docker naming
DOCKER in names is confusingly used as synonym for "image", "container",
and "ci". Fix the confusion by picking the term that fits the context.

-BEGIN VERIFY SCRIPT-
 ren() { sed -i "s:$1:$2:g" $( git grep -l "$1" ) ; }

 ren DOCKER_PACKAGES CI_BASE_PACKAGES
 # This better reflects that they are the common base for all CI
 # containers.

 ren DOCKER_ID CI_CONTAINER_ID
 # This is according to the documentation of "--detach , -d: Run
 # container in background and print container ID".

 ren DOCKER_NAME_TAG CI_IMAGE_NAME_TAG
 # This avoids confusing with CONTAINER_NAME and clarifies that it is an
 # image.

 ren DOCKER_ADMIN CI_CONTAINER_CAP
 # This clarifies that it is a capability added to the container.

 ren DOCKER_CI_CMD_PREFIX CI_EXEC_CMD_PREFIX
 # This brings it in line with the CI_EXEC naming.

-END VERIFY SCRIPT-
2023-01-11 10:49:18 +01:00
MarcoFalke fadfae42f1
ci: Remove unused busybox workaround
The find workaround is no longer needed after commit
d3d547c545
2023-01-09 12:40:45 +01:00
MarcoFalke fac424fce7
ci: Create named symbol for BINS_SCRATCH_DIR
Also, create the dir a bit earlier, right after it was put in the PATH.
2023-01-09 11:53:20 +01:00
MarcoFalke fa4e98c77f
ci: Fix ci_native_fuzz_msan CONTAINER_NAME 2023-01-07 11:47:38 +01:00
Hennadii Stepanov 1a0d8e178c
build: Re-enable external signer on Windows 2023-01-06 10:53:36 +00:00
MarcoFalke bf3b589413
Merge bitcoin/bitcoin#26791: ci: Properly set COMMIT_RANGE in lint task
fa5cbf2290 ci: Properly set COMMIT_RANGE in lint task (MarcoFalke)

Pull request description:

  Currently the variable holds (apart from the commits in the pull request) all commits to master since the pull was opened.

  This is problematic, because already merged commits are linted in unrelated pulls, leading to:

  * Wasted resources. For example, currently the lint task may take 9 minutes, when it should take 1. See https://cirrus-ci.com/task/6032782770569216?logs=lint#L1449
  * False failures. For example, when a "wrong" commit is in master it can lead to some pulls failing unrelatedly, and others not.

  Now that the CI has the `/merge` commit (since commit fad7281d78), `COMMIT_RANGE` can simply be set to `HEAD~..HEAD` to only hold the changes in the pull.

ACKs for top commit:
  fanquake:
    ACK fa5cbf2290

Tree-SHA512: e85fca4ca9d2615ddd2544403485e06885769a3f70bca297e23eefda2a1d28f47c5271f6adfa6ce0e5e972335c78098b76e0db4b109f59d0986bf508cef7528f
2023-01-04 13:56:45 +01:00
Jon Atack 1e5e87cec3 script: update python linter dependencies 2023-01-03 11:05:09 -08:00
MarcoFalke fa5cbf2290
ci: Properly set COMMIT_RANGE in lint task 2023-01-02 14:05:23 +01:00
MarcoFalke d8bdee0fc8
Merge bitcoin/bitcoin#26775: ci: Revert tsan task changes
fabb6af850 ci: Remove duplicate CC and CXX from tsan task (MarcoFalke)
fa5d9a0e24 Revert "ci: Use clang-15 in tsan task" (MarcoFalke)
faa835e7e5 Revert "test: Drop no longer needed `race:epoll_ctl` TSan suppression" (MarcoFalke)

Pull request description:

  Looks like there are still bugs in clang-15, so we need to roll back all the way to the previously used version (clang-13).

ACKs for top commit:
  hebasto:
    ACK fabb6af850, I have reviewed the code and it looks OK, I agree it can be merged.

Tree-SHA512: d62203049847ab9095ee3fc89e18bdd721d1d9d5a7ef7a9f524c80e6be58d1d9f6aa2f14533df1ea77eb59597fba6fa9b987b17eb03b2c3f7cb577ab59cd59c0
2023-01-01 10:26:02 +01:00
MarcoFalke fabb6af850
ci: Remove duplicate CC and CXX from tsan task 2022-12-30 09:50:22 +01:00
MarcoFalke fa5d9a0e24
Revert "ci: Use clang-15 in tsan task"
This reverts commit faa00ca78e.
2022-12-30 09:49:51 +01:00
MarcoFalke 65de8eeeca
Merge bitcoin/bitcoin#26770: ci: Move `--enable-c++20` from "tidy" task back to "ASan..." one
afc6052430 ci: Move `--enable-c++20` from "tidy" task back to "ASan..." one (Hennadii Stepanov)

Pull request description:

  This PR reverts cc7335edc8 from https://github.com/bitcoin/bitcoin/pull/25528 partially.

  C++20 has introduced some new headers, and it is premature to consider them when using the IWYU tool.

  Required for https://github.com/bitcoin/bitcoin/pull/26763 and https://github.com/bitcoin/bitcoin/pull/26766.

  Related discussions:
  - https://github.com/bitcoin/bitcoin/pull/25528#discussion_r1058906785
  - https://github.com/bitcoin/bitcoin/pull/26763#discussion_r1058860943

ACKs for top commit:
  MarcoFalke:
    review only ACK afc6052430

Tree-SHA512: 9c1d3317d0f7a94d46d1e442da1a91669cd9ed1e62a41579edc96e1f5447551b536d32eeb222caf277e85228482753be545e0a874208cb24f9a8491fce3b6d9f
2022-12-29 20:49:21 +01:00
Hennadii Stepanov afc6052430
ci: Move `--enable-c++20` from "tidy" task back to "ASan..." one
This change reverts cc7335edc8 partially.

C++20 has introduced some new headers, and it is premature to consider
them when using the IWYU tool.
2022-12-29 18:39:00 +00:00
MarcoFalke faa00ca78e
ci: Use clang-15 in tsan task 2022-12-29 16:30:13 +01:00
Hennadii Stepanov 306ccd4927
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-

Commits of previous years:
- 2021: f47dda2c58
- 2020: fa0074e2d8
- 2019: aaaaad6ac9
2022-12-24 23:49:50 +00:00
fanquake d3a84347e8
ci: remove --prefix from msan job 2022-12-20 17:17:35 +00:00
pasta 935acdcc79
refactor: modernize the implementation of uint256.*
- Constructors of uint256 to utilize Span instead of requiring a std::vector
- converts m_data into a std::array
- Prefers using `WIDTH` instead of `sizeof(m_data)`
- make all the things constexpr
- replace C style functions with c++ equivalents
    - memset -> std::fill
    - memcpy -> std::copy
        Note: In practice, implementations of std::copy avoid multiple assignments and use bulk copy functions such as std::memmove if the value type is TriviallyCopyable and the iterator types satisfy LegacyContiguousIterator. (https://en.cppreference.com/w/cpp/algorithm/copy)
    - memcmp -> std::memcmp
2022-12-10 14:34:44 -06:00
Hennadii Stepanov 574e50addf
ci: Use `CONFIG_SITE` variable and `--prefix` option properly
This change fixes scripts when they are being run locally with a pre-
existed `$DEPENDS_DIR/$HOST` directory.
2022-12-10 19:13:20 +00:00
MarcoFalke 9e59d21fbe
Merge bitcoin/bitcoin#26592: ci: only run USDT interface tests on CirrusCI
2811f40f30 ci: only run USDT interface tests on CirrusCI (0xb10c)

Pull request description:

  As mentioned in #26571, the task running the USDT interface tests fail when run in docker. cc7335edc8 in #25528 added that the tests are run in a **VM** in Cirrus CI. Running them locally in docker containers might not work:

  - We use [bcc] as tracing toolkit which requires the kernel headers to compile the BPF bytecode. As docker containers use the hosts kernel and don't run their own, there is a potential for mismatches between kernel headers available in the container and the host kernel. This results in a failure loading the BPF byte code.
  - Privilges are required to load the BPF byte code into the kernel. Normally, the docker containers aren't run with these.
  - We currently use an untrusted third-party PPA to install the bpfcc-tools package on Ubuntu 22.04. Using this on a local dev system could be a security risk.

  To not hinder the ASan + LSan + UBSan part of the CI task, the USDT tests are disabled on non-CirrusCI runs.

  [bcc]: https://github.com/iovisor/bcc

Top commit has no ACKs.

Tree-SHA512: 7c159b59630b36d5ed927b501fa0ad6f54ff3d98d0902f975cc8122b4147a7f828175807d40a470a85f0f3b6eeb79307d3465a287dbd2f3d75b803769ad0b6e7
2022-12-02 10:52:52 +01:00
0xb10c 2811f40f30
ci: only run USDT interface tests on CirrusCI
As mentioned in #26571, the task running the USDT interface tests
fail when run in docker. cc7335edc8
in #25528 added that the tests are run in a **VM** in Cirrus CI.
Running them locally in docker containers might not work:

- We use [bcc] as tracing toolkit which requires the kernel headers
  to compile the BPF bytecode. As docker containers use the hosts
  kernel and don't run their own, there is a potential for mismatches
  between kernel headers available in the container and the host
  kernel. This results in a failure loading the BPF byte code.
- Privilges are required to load the BPF byte code into the kernel.
  Normally, the docker containers aren't run with these.
- We currently use an untrusted third-party PPA to install the
  bpfcc-tools package on Ubuntu 22.04. Using this on a local dev
  system could be a security risk.

To not hinder the ASan + LSan + UBSan part of the CI task, the USDT
tests are disabled on non-CirrusCI runs.

[bcc]: https://github.com/iovisor/bcc
2022-11-28 21:26:26 +01:00
MarcoFalke d415b7261c
Merge bitcoin/bitcoin#26588: ci: Skip COMMIT_RANGE if no CIRRUS_PR
fad1c55301 lint: Skip COMMIT_RANGE if no CIRRUS_PR (MarcoFalke)

Pull request description:

  It doesn't make sense to run this for non-PRs, because:

  * There are known whitespace "violations" in previous commits, so the lint may fail
  * Once the changes are merged, it is too late to fix them up (force pushes are illegal)
  * It isn't possible to determine which commits to run on if there is no reference branch (target branch of the pull request)

  Moreover, the test fails on non-master:

  * https://github.com/bitcoin/bitcoin/runs/8664441400

  Fix all issues by skipping it.

ACKs for top commit:
  hebasto:
    ACK fad1c55301, also tested in my personal Cirrus account.

Tree-SHA512: be15f00e2b2a9069583833545883e0e5968a33d2455dad59e6fb47c1102b4dd16ef932e9ba945e29e9d941e6c17bd531a02c66b0491097801be6bda476875537
2022-11-28 17:18:29 +01:00
MarcoFalke fad1c55301
lint: Skip COMMIT_RANGE if no CIRRUS_PR 2022-11-28 11:09:30 +01:00
MarcoFalke c239d3dac9
Merge bitcoin/bitcoin#26574: ci: use ci_exec_root for clang install
54dd8f51ce ci: use ci_exec_root for clang install (josibake)

Pull request description:

  fixes a bug introduced in #25900 ; see https://github.com/bitcoin/bitcoin/pull/25900#issuecomment-1327311069

  the general idea of #25900 was to use a non-root user as much as possible to avoid modifying the user's local filesystem. however, it appears the root user is needed to correctly install clang.

ACKs for top commit:
  hebasto:
    ACK 54dd8f51ce, tested on Ubuntu 22.04.

Tree-SHA512: beb01d4b6127fbba3c8d18e85cf7ec7d1b2ec93ea05c475ab51bcaa04ef1b0591d886f1a7e0732c5ae86806013f022c0b44027380d2b0cfb1bfdc843e40f99b4
2022-11-28 10:20:21 +01:00
josibake 54dd8f51ce
ci: use ci_exec_root for clang install 2022-11-25 14:13:29 +01:00
MarcoFalke fa3b2cf277
fuzz: Move-only net utils 2022-11-23 17:26:01 +01:00
MarcoFalke 85892f77c9
Merge bitcoin/bitcoin#25900: ci: run docker wrapper with a non-root user
849f20a6d3 ci: create and use non-root user for docker image (josibake)

Pull request description:

  Previously, everything in the ci docker image ran as the root user. This would lead to certain directories (`ci/scratch`, `depends`) being owned by `root` after running the ci locally which would lead to annoying behavior such as subsequent guix builds failing due to `depends/` being owned by root.

  This PR adds a non-root user in the container and chowns the mounted working directory. All the `docker exec` commands now run as the non-root user, except for the few that still need to run as root (mainly, installing packages).

  To test this I checked out a fresh copy of the repo, applied my changes, ran the CI, and verified all the local file permissions were unchanged after the CI was finished running.

ACKs for top commit:
  hebasto:
    ACK 849f20a6d3, tested on Ubuntu 22.04 by running commands as follows:

Tree-SHA512: 734dca0f36157fce5fab243b4ff657fc17ba980e8e4e4644305f41002ff21bd5cef02c306ea1e0b5c841d4c07c095e8e4be16722e6a38c890717c60a3f5ec62a
2022-11-22 12:46:40 +01:00
fanquake 1b680948d4
Merge bitcoin/bitcoin#26292: util: move threadinterrupt into util/
b89530483d util: move threadinterrupt into util (fanquake)

Pull request description:

  Alongside thread and threadnames. It's part of libbitcoin_util.

ACKs for top commit:
  ryanofsky:
    Code review ACK b89530483d. No changes since last review other than rebase
  theuni:
    ACK b89530483d.

Tree-SHA512: 0421f4d1881ec295272446804b27d16bf63e6b62b272f8bb52bfecde9ae6605e8109ed16294690d3e3ce4b15cc5e7c4046f99442df73adb10bdf069d3fb165aa
2022-11-22 09:52:53 +00:00
josibake 849f20a6d3
ci: create and use non-root user for docker image
Running all commands as the root user in the docker image
will change local file permissions in the ci and depends directory.

Add a non-root user to the container and use this user whenever
possible when running docker exec commands.
2022-11-21 18:11:28 +01:00
MacroFake 2222ec71fd
util: Move error message formatting of NonFatalCheckError to cpp
This allows to strip down the header file
2022-11-16 12:21:33 +01:00
fanquake b89530483d
util: move threadinterrupt into util 2022-11-01 10:14:49 +00:00
fanquake 3a0b352c63
refactor: move url.h/cpp from lib util to lib common 2022-10-31 10:17:04 +00:00
MacroFake bd478890c5
Merge bitcoin/bitcoin#26388: ci: Use `macos-ventura-xcode:14.1` image for "macOS native" task
da16893474 ci: Use `macos-ventura-xcode:14.1` image for "macOS native" task (Hennadii Stepanov)
702836530f ci: Make `getopt` path architecture agnostic (Hennadii Stepanov)

Pull request description:

  The "macOS native" CI task always uses the recent OS image.

  This PR updates it up to the recent macOS release.

  Cirrus Labs [stopped](https://github.com/bitcoin/bitcoin/pull/25160#issuecomment-1162829773) updating macOS images for `x86_64`, therefore, an `arm64` image been used.

  Also `make test-security-check` has been dropped as it ["isn't even expected to pass"](https://github.com/bitcoin/bitcoin/issues/26386#issuecomment-1290318628) on `arm64` in CI.

ACKs for top commit:
  Sjors:
    utACK da16893

Tree-SHA512: 36785d33b7f11b3cdbc53bcfbf97d88bf821fad248c825982dd9f8e3413809a4ef11190eaf950e60fdf479b62ff66920c35d9ea42d534723f015742eec7e19b6
2022-10-27 16:15:20 +02:00
Hennadii Stepanov da16893474
ci: Use `macos-ventura-xcode:14.1` image for "macOS native" task 2022-10-25 13:39:03 +01:00
Hennadii Stepanov 702836530f
ci: Make `getopt` path architecture agnostic 2022-10-25 09:49:07 +01:00
MacroFake aaaa7bd0ba
iwyu: Add missing includes 2022-10-18 14:12:52 +02:00
MacroFake 9ca39d69df
Merge bitcoin/bitcoin#26254: iwyu: Add zmq source files
13afcc0cd4 iwyu: Add zmq source files (Hennadii Stepanov)

Pull request description:

ACKs for top commit:
  fanquake:
    ACK 13afcc0cd4

Tree-SHA512: 7af95e991fc2782aeba2edfef0a2f75f9c361058295586adb062087aa31c47cfcce2425aee9dd5153e18e018cf1f9272c9617c671b7262db55f241526c3fcb15
2022-10-10 18:08:45 +02:00
Hennadii Stepanov 13afcc0cd4
iwyu: Add zmq source files 2022-10-10 15:44:02 +01:00
MacroFake fa04376554
Remove clang-format from lint task
clang-format could be used in scripted diffs, but remained largely
unused.
2022-10-05 10:52:42 +02:00
fanquake a23f8c8978
Merge bitcoin/bitcoin#26234: ci: Allow PIP_PACKAGES on centos
fa6054e952 ci: Allow PIP_PACKAGES on centos (MacroFake)
fac085a05c ci: Remove unused package (MacroFake)

Pull request description:

  This was added in 7fc5e865b9 but I can't see a reason why this should be forbidden.

  This is also needed for other changes (bumping the minimum python version).

ACKs for top commit:
  hebasto:
    re-ACK fa6054e952

Tree-SHA512: e8ead9ee00079024eb1e8c6e7b31c78cf2a3392159b444765c2ea9a58bed2a7550bf71083210692a45bb8ed7896cb882b72bf70baa13a2384864b2b510b73005
2022-10-04 21:39:29 +01:00
MacroFake fa6054e952
ci: Allow PIP_PACKAGES on centos
This was added in 7fc5e865b9 but I can't
see a reason why this should be forbidden.
2022-10-04 11:53:31 +02:00
MacroFake fac085a05c
ci: Remove unused package
Address feedback from https://github.com/bitcoin/bitcoin/pull/24561/files#r985719812
2022-10-04 11:51:33 +02:00
Hennadii Stepanov ac1d99240a
ci: Move `git config` commands into script where they are used 2022-10-04 08:51:55 +01:00
MacroFake d8ded8bc08 ci: Use git2.34 for lint task 2022-09-28 19:58:13 +02:00
Jon Atack 8f2010de6e Bump codespell version to 2.2.1 2022-09-15 12:53:32 +02:00
MacroFake fa875349e2
Fix iwyu 2022-08-20 09:33:01 +02:00
MacroFake 7d82f86341
Merge bitcoin/bitcoin#25650: script: default to necessary tags in `test/get_previous_releases.py`
21a9e94dbb ci: remove hardcoded tag list from ci scripts (josibake)
d530ba390e doc: update test/README.md (josibake)
614d4682ba script: default to necessary tags in get_previous_releases.py (josibake)

Pull request description:

  Almost every time I need to use this script, I forget the tag list is needed and that a specific set of tags is needed for the backwards compatibility tests to work. I end up wasting time reading through the script and googling to find the tag list before remembering it is in `test/README.md`

  I assume (hope) I'm not the only one this happens to, so I figured it would make more sense to have the script default to downloading/building the necessary tags. This has the added benefit of making the script the source of truth: the script already needs to be updated with the SHA256_SUM of the binary for every new tag that is added, so it makes sense to use `SHA256_SUMS` list as the necessary tag list. This means there is less risk of the README and the script drifting (i.e updating the readme with a new tag and forgetting to update the script, or updating the script and forgetting to update the README). Now all that needs to happen is to update the `SHA256_SUMS` list in the script and everything Just Works (TM)

ACKs for top commit:
  Sjors:
    re-tACK 21a9e94dbb

Tree-SHA512: 97b488227a89a6827584edd251820a7074fad75dfd7f26f1aa5f858e2521d2e02effd0f11e6dc4676e1155d3d5aba6ff94a4b58ffef80dc201376afd5927deb9
2022-08-05 10:51:06 +02:00
Carl Dong 41c5201a90 validationcaches: Add and use ValidationCacheSizes
Also:

- Make DEFAULT_MAX_SIG_CACHE_SIZE into constexpr
  DEFAULT_MAX_SIG_CACHE_BYTES to utilize the compile-time integer
  arithmetic overflow checking available to constexpr.
- Fix comment (MiB instead of MB) for DEFAULT_MAX_SIG_CACHE_BYTES.
- Pass in max_size_bytes parameter to InitS*Cache(), modify log line to
  no longer allude to maxsigcachesize being split evenly between the two
  validation caches.
- Fix possible integer truncation and add a comment.

[META] I've kept the integer types as int64_t in order to not introduce
       unintended behaviour changes, in the next commit we will make
       them size_t.
2022-08-03 12:03:27 -04:00
MacroFake ddddd6913b
sort after scripted-diff 2022-08-02 15:31:05 +02:00
MacroFake fac812ca83
scripted-diff: Move mempool_args to src/node
It is part of the node library. Also, it won't be moved to the kernel
lib, as it will be pruned of ArgsManager.

-BEGIN VERIFY SCRIPT-
 # Move module
 git mv src/mempool_args.cpp src/node/
 git mv src/mempool_args.h   src/node/
 # Replacements
 sed -i 's:mempool_args\.h:node/mempool_args.h:g'     $(git grep -l mempool_args)
 sed -i 's:mempool_args\.cpp:node/mempool_args.cpp:g' $(git grep -l mempool_args)
 sed -i 's:MEMPOOL_ARGS_H:NODE_MEMPOOL_ARGS_H:g'      $(git grep -l MEMPOOL_ARGS_H)
-END VERIFY SCRIPT-
2022-08-02 15:31:01 +02:00
MacroFake fa468bdfb6
Return optional error from ApplyArgsManOptions
Also pass in a (for now unused) reference to the params.

Both changes are needed for the next commit.
2022-08-02 15:21:50 +02:00
MacroFake eeb5a94e27
Merge bitcoin/bitcoin#25528: ci: run USDT interface tests in the CI
cc7335edc8 ci: run USDT interface test in a VM (0xb10c)
dba6f82342 test: adopt USDT utxocache interface tests (0xb10c)
220a5a2841 test: hook into PID in tracing tests (0xb10c)

Pull request description:

  Changes a CI task that runs test the previously not run `test/functional/interface_usdt_*.py` functional tests (added in https://github.com/bitcoin/bitcoin/pull/24358).

  This task is run as CirussCI `compute_engine_instance` VM as hooking into the tracepoints is not possible in CirrusCI docker containers (https://github.com/bitcoin/bitcoin/issues/23296#issuecomment-1024920845). We use an unoffical PPA and untrusted  `bpfcc-tools` package in the CI as the Ubuntu jammy and Debian bullseye packages are outdated. We hope use an official package when new Ubuntu/Debian releases are available for the use with Google Compute Engine.

  We make sure to hook into `bitcoind` binaries in USDT interface tests via their PID, instead of their path. This makes sure multiple functional tests running in parallel don't interfere with each other.

  The utxocache USDT interface tests is adopted to a change of the functional test framework that wasn't detected as the tests weren't run in the CI. As the tracepoints expose internals, it can happen that we need to adopt the interface test when internals change. This is a bit awkward, and if it happens to frequently, we should consider generalizing the tests a bit more. For now it's fine, I think.

  See the individual commit messages for more details on the changes.

  Fixes https://github.com/bitcoin/bitcoin/issues/24782
  Fixes https://github.com/bitcoin/bitcoin/issues/23296

  I'd like to hear from reviewers:
  - Are we OK with using the [`hadret/bpfcc`](https://launchpad.net/~hadret/+archive/ubuntu/bpfcc) PPA for now? There is a clear plan when to drop it and as is currently, it could only impact the newly added VM task.
  - ~~Adding a new task increases CI runtime and costs. Should an existing `container` CI task be ported to a VM and reused instead?~~ Yes, see https://github.com/bitcoin/bitcoin/pull/25528#issuecomment-1179509525

ACKs for top commit:
  MarcoFalke:
    cr ACK cc7335edc8

Tree-SHA512: b7fddccc0a77d82371229d048abe0bf2c4ccaa45906497ef3040cf99e7f05561890aef4c253c40e4afc96bb838c9787fae81c8454c6fd9db583276e005a4ccb3
2022-08-01 11:27:29 +02:00
fanquake 3fe58a98d3
tidy: run clang-tidy in quiet mode 2022-07-29 16:12:45 +01:00
fanquake 5671217477
Merge bitcoin/bitcoin#24974: refactor: Make FEELER_SLEEP_WINDOW type safe (std::chrono)
fa74e726c4 refactor: Make FEELER_SLEEP_WINDOW type safe (std::chrono) (MacroFake)
fa3b3cb9b5 Expose underlying clock in CThreadInterrupt (MacroFake)

Pull request description:

  This gets rid of the `value*1000` manual conversion.

ACKs for top commit:
  naumenkogs:
    utACK fa74e726c4
  dergoegge:
    Code review ACK fa74e726c4

Tree-SHA512: 90409c05c25f0dd2f1c4dead78f707ebfd78b7d84ea4db9fcefd9c4958a1a3338ac657cd9e99eb8b47d52d4485fa3c947dce4ee1559fb56ae65878685e1ed9a3
2022-07-26 15:09:21 +01:00
MacroFake fa2247a9f9
refactor: Make CTransaction constructor explicit
It involves calculating two hashes, so the performance impact should be
made explicit.

Also, add the module to iwyu.
2022-07-25 12:16:54 +02:00
josibake 21a9e94dbb
ci: remove hardcoded tag list from ci scripts 2022-07-21 12:02:08 +02:00
Hennadii Stepanov 6d58117a31
build: Build minisketch test in `make check`, not in `make` 2022-07-21 10:42:09 +01:00
fanquake cc5b39e44e
ci: better pin to dwarf4 in valgrind job
Use `-gdwarf` and also set CFLAGS. I was seeing Valgrind issues otherwise.
2022-07-21 10:16:46 +01:00
Ryan Ofsky 1e761a0169 ci: Enable IWYU in src/kernel directory
Suggested https://github.com/bitcoin/bitcoin/pull/25308#discussion_r892505713
2022-07-19 16:54:52 -04:00
MacroFake 1111ddeedf
Remove unused includes from dbwrapper.h 2022-07-19 14:32:53 +02:00
MacroFake fa869ce2c2
Add missing includes to node/chainstate
This is needed for the next commit
2022-07-19 14:12:14 +02:00
glozow 821f5c824f
Merge bitcoin/bitcoin#25487: [kernel 3b/n] Decouple `{Dump,Load}Mempool` from `ArgsManager`
cb3e9a1e3f Move {Load,Dump}Mempool to kernel namespace (Carl Dong)
aa30676541 Move DEFAULT_PERSIST_MEMPOOL out of libbitcoinkernel (Carl Dong)
06b88ffb8a LoadMempool: Pass in load_path, stop using gArgs (Carl Dong)
b857ac60d9 test/fuzz: Invoke LoadMempool via CChainState (Carl Dong)
b3267258b0 Move FopenFn to fsbridge namespace (Carl Dong)
ae1e8e3756 mempool: Use NodeClock+friends for LoadMempool (Carl Dong)
f9e8e5719f mempool: Improve comments for [GS]etLoadTried (Carl Dong)
813962da0b scripted-diff: Rename m_is_loaded -> m_load_tried (Carl Dong)
413f4bb52b DumpMempool: Pass in dump_path, stop using gArgs (Carl Dong)
bd4407817e DumpMempool: Use std::chrono instead of weird int64_t arthmetics (Carl Dong)
c84390b741 test/mempool_persist: Test manual savemempool when -persistmempool=0 (Carl Dong)

Pull request description:

  This is part of the `libbitcoinkernel` project: #24303, https://github.com/bitcoin/bitcoin/projects/18

  -----

  This PR moves `{Dump,Load}Mempool` into its own `kernel/mempool_persist` module and introduces `ArgsManager` `node::` helpers in `node/mempool_persist_args`to remove the scattered calls to `GetBoolArg("-persistmempool", DEFAULT_PERSIST_MEMPOOL)`.

  More context can be gleaned from the commit messages.

  -----

  One thing I was reflecting on as I wrote this was that in the long run, I think we should probably invert the validation <-> mempool relationship. Instead of mempool not depending on validation, it might make more sense to have validation not depend on mempool. Not super urgent since `libbitcoinkernel` will include both validation and mempool, but perhaps something for the future.

ACKs for top commit:
  glozow:
    re ACK cb3e9a1e3f via `git range-diff 7ae032e...cb3e9a1`
  MarcoFalke:
    ACK cb3e9a1e3f 🔒
  ryanofsky:
    Code review ACK cb3e9a1e3f

Tree-SHA512: 979d7237c3abb5a1dd9b5ad3dbf3b954f906a6d8320ed7b923557f41a4472deccae3e8a6bca0018c8e7a3c4a93afecc502acd1e26756f2054f157f1c0edd939d
2022-07-18 16:09:27 +01:00
fanquake 70d807c355
refactor: integrate no_nul into univalue unitester 2022-07-18 09:15:18 +01:00
Carl Dong cb3e9a1e3f Move {Load,Dump}Mempool to kernel namespace
Also:
1. Add the newly introduced kernel/mempool_persist.cpp to IWYU CI script
2. Add chrono mapping for iwyu
2022-07-15 12:26:20 -04:00
MacroFake fa3b3cb9b5
Expose underlying clock in CThreadInterrupt
Overloading sleep_for is not needed, as

* seconds and minutes can be converted to milliseconds by the compiler,
  not needing a duration_cast
* std::condition_variable::wait_for will convert milliseconds to the
  duration type of the underlying clock

So simply expose the clock.
2022-07-13 15:20:49 +02:00
0xb10c cc7335edc8
ci: run USDT interface test in a VM
Our CI tasks are run by CirrusCI in Docker containers in a Google
Compute Engine based Kubernetes environment. These containers have
limited capabilities - especially CAP_SYS_ADMIN is missing. See
https://github.com/bitcoin/bitcoin/issues/23296#issuecomment-1024920845

We need elevated privileges to hook into the USDT tracepoints. We use a
CirrusCI "compute_engine_instance" (a VM, not a container) where we have
the required privileges. The ubunut-mininmal-2204-lts was choosen with
debian-11 being an alternative. Both pack an outdated 'bpfcc-tools'
package (v0.18.0) from 2020. This version prints warnings to stderr
during BPF bytecode compilation, which causes our functional test runner
to fail. This is fixed in newer verison.

Until debian-12 or a newer Ubuntu release is avaliable as image in GCE
(https://cloud.google.com/compute/docs/images/os-details), we use a
third-party and untrusted PPA that releases up-to-date versions of the
package.

The official iovisor (authors of BCC) PPA is outdated too. An
alternative would be to compile BCC from source in the CI.

Co-authored-by: MacroFake <falke.marco@gmail.com>
2022-07-08 19:41:58 +02:00
fanquake 07f2c25d04
refactor: add most of src/util to iwyu
These files change infrequently, and not much header shuffling is required.

We don't add everything in src/util/ yet, because IWYU makes some
dubious suggestions, which I'm going to follow up with upstream.
2022-07-08 11:06:01 +01:00
MacroFake bac52a1cfe
Merge bitcoin/bitcoin#25447: fuzz: add low-level target for txorphanage
6eb0909cb7 fuzz: add low-level target for txorphanage (chinggg)

Pull request description:

  This adds a low-level fuzz target for orphan transaction handling by creating random transactions and calling all functions in `TxOrphanage`.
  It cannot simulate real-world `orphan/unorphan` scenarios effectively since it does not maintain any state about the node and the chain. A high-level fuzz target which construct well-designed transaction graphs will be added later.

ACKs for top commit:
  MarcoFalke:
    review ACK 6eb0909cb7 🐈

Tree-SHA512: b4d64f5941df77d13981f75ec170cef6ffabe782797c982ede7f34134be01dc0026dd7c0bee614bc1d64715e90a933d2a8c95974d402e32eaba8e24cc928299e
2022-07-07 14:03:52 +02:00
chinggg 6eb0909cb7 fuzz: add low-level target for txorphanage 2022-07-06 22:13:54 +08:00
sogoagain 31346a3196 [ci] apply cache size limit and print ccache statistics in "ARM64 Android APK" 2022-07-04 22:18:41 +09:00
Hennadii Stepanov 0bb7a1f71d
ci: Improve naming related to "macOS 12 native x86_64" task 2022-06-22 10:15:16 +02:00
fanquake d873ff96e5
refactor: cleanups post unsubtree'ing univalue
Mostly changes to remove src/univalue exceptions from the various linters,
and the required code changes to make them happy. As well as minor doc
changes.
2022-06-15 12:56:44 +01:00
MacroFake 2ab4a80480
Merge bitcoin/bitcoin#25254: Move minRelayTxFee to policy/settings
fa4068b4e2 Move minRelayTxFee to policy/settings (MacroFake)

Pull request description:

  Seems a bit confusing to put policy stuff into validation, so fix that.

  Also fix includes via `iwyu`.

ACKs for top commit:
  ariard:
    ACK fa4068b, the includes move compiles well locally.
  ryanofsky:
    Code review ACK fa4068b4e2. Make sense to move the global variable to policy/settings and the default constant to policy/policy. Ariard points out other constants that could be moved, which seems fine, but it seems like moving the global variable to be with other related global variables is more significant.

Tree-SHA512: adf9619002610d1877f3aef0a9e6115fc4c2ad64135a3e5100824c650b560c47f47ac28894c6214a50a7888355252a9f6f7cec98c23a771a1964160ef1ca77de
2022-06-07 11:31:10 +02:00
MacroFake f66633d9cb
Merge bitcoin/bitcoin#25288: test: Reliably don't start itself (lint-all.py runs all tests twice)
f26a496dfd test: clean up all-lint.py (Martin Leitner-Ankerl)
64d72c4c87 test: rename lint-all.py to all-lint.py (Martin Leitner-Ankerl)

Pull request description:

  When running `./test/lint/lint-all.py`, the script runs all tests but
  also calls itself because the comparison with `__file__` doesn't work.

  Comparing resolved paths gives reliable comparison, and lint-all.py doesn't call itself any more

ACKs for top commit:
  laanwj:
    Code review ACK f26a496dfd

Tree-SHA512: b44abdd685f7b48a6a9f48e96d97138b635c31c1c7ab543cb5636b5f49690ccd56fa6fec01ae7fcc16af01a613372ee77632f70c32059919b373aa8051953791
2022-06-07 10:37:34 +02:00
Martin Leitner-Ankerl 64d72c4c87 test: rename lint-all.py to all-lint.py
That way it is impossible for the script to call itself.
2022-06-07 10:22:45 +02:00
Hennadii Stepanov c47944f4e9
ci: Reuse some configure options in "ARM64 Android APK" task 2022-05-31 18:49:53 +02:00
Hennadii Stepanov 7739438811
ci, android: Update NDK up to r23c 2022-05-31 18:45:13 +02:00
Hennadii Stepanov ca0c3e5077
ci, android: Update Command-line Tools from 2.1 up to 7.0 2022-05-31 18:43:11 +02:00
Hennadii Stepanov 8790da3c1e
ci: Drop unneeded packages in "ARM64 Android APK" task 2022-05-31 15:48:25 +02:00
MacroFake fa4068b4e2
Move minRelayTxFee to policy/settings
Also fix includes using iwyu
2022-05-31 15:05:57 +02:00
Hennadii Stepanov 3ffc190321
ci: Install documented packages for "Win64" CI task 2022-05-28 11:34:11 +02:00
Jon Atack 345647c4da ci: add DEBUG_LOCKCONTENTION to CI task containing DEBUG_LOCKORDER 2022-05-20 12:48:17 +02:00
Sjors Provoost dba1231672
test: previous releases: add v23.0
Starting from v23.0 there is a separate macOS release for x86_64 and aarch64.

Co-authored-by: Pavol Rusnak <pavol@rusnak.io>
2022-05-06 10:00:47 +02:00
MacroFake 2c56404088
Merge bitcoin/bitcoin#25029: rpc: Move fee estimation RPCs to separate file
fa753abd7c rpc: Move fee estimation RPCs to separate file (MacroFake)

Pull request description:

  Fee estimation is generally used by wallets when creating txs. It doesn't have anything to do with creating or submitting blocks.

ACKs for top commit:
  pk-b2:
    ACK fa753abd7c
  brunoerg:
    crACK fa753abd7c

Tree-SHA512: 81e0edc936198a0baf0f5bfa8cfedc12db51759c7873bb0082dfc5f0040d7f275b35f639c6f5b86fa1ea03397b0d5e757c2ce1b6b16f1029880a39b9c3aaceda
2022-05-03 08:17:50 +02:00
MacroFake fa847ed2f6
ci: Clone iwyu only if missing 2022-04-30 09:23:56 +02:00
MacroFake fa753abd7c
rpc: Move fee estimation RPCs to separate file
Can be reviewed with --color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
2022-04-29 16:25:06 +02:00
fanquake 91ac12be44
Merge bitcoin/bitcoin#25013: Remove cs_main from verifymessage, move msg utils to new file
fa60169811 rpc: Move signmessage RPC util to new file (MacroFake)
fa9425177e Remove cs_main from verifymessage (MacroFake)

Pull request description:

  The `verifymessage` RPC has several issues:

  * It takes `cs_main` for no reason, blocking progress on removing the `cs_main` global mutex.
  * It is located in a file called `misc`, which is not a very helpful name.

  Fix all issues.

ACKs for top commit:
  vincenzopalazzo:
    ACK fa60169811

Tree-SHA512: c71a1f481b828e0a544405fecbbc7ca44e66ea46b498d7aed1f1c584d6a99724deb13e89d90b9d5cdeecbce293e6a41e9f7ae299543f6d761bf9e7a839b6c7f3
2022-04-29 12:20:10 +01:00
fanquake dd17c42a16
Merge bitcoin/bitcoin#24322: [kernel 1/n] Introduce initial `libbitcoinkernel`
035fa1f07a build: Remove LIBTOOL_APP_LDFLAGS for bitcoin-chainstate (Cory Fields)
3f0595095d docs: Add libbitcoinkernel_la_SOURCES explanation (Carl Dong)
94ad45deb2 ci: Build libbitcoinkernel (Carl Dong)
26b2e7ffb3 build: Extract the libbitcoinkernel library (Carl Dong)
1df44dd20c b-cs: Define G_TRANSLATION_FUN in bitcoinkernel.cpp (Carl Dong)
83a0bb7cc9 build: Separate lib_LTLIBRARIES initialization (Carl Dong)
c1e16cb31f build: Create .la library for bitcoincrypto (Carl Dong)
8bdfe057c7 build: Create .la library for leveldb (Carl Dong)
05d1525b6d build: Create .la library for crc32c (Carl Dong)
64caf94479 build: Remove vestigial LIBLEVELDB_SSE42 (Carl Dong)
1392e8e2d8 build: Don't add unrelated libs to LIBTEST_* (Carl Dong)

Pull request description:

  Part of: #24303

  This PR introduces a `libbitcoinkernel` static library linking in the minimal list of files necessary to use our consensus engine as-is. `bitcoin-chainstate` introduced in #24304 now will link against `libbitcoinkernel`.

  Most of the changes are related to the build system.

  Please read the commit messages for more details.

ACKs for top commit:
  theuni:
    This may be my favorite PR ever. It's a privilege to ACK 035fa1f07a.

Tree-SHA512: b755edc3471c7c1098847e9b16ab182a6abb7582563d9da516de376a770ac7543c6fdb24238ddd4d3d2d458f905a0c0614b8667aab182aa7e6b80c1cca7090bc
2022-04-28 15:14:32 +01:00
laanwj 85aea18ae6
Merge bitcoin/bitcoin#24982: tests: Port `lint-all.sh` to `lint-all.py`
29f44fed36 Converting `lint-all.sh` to `lint-all.py`. (hiago)

Pull request description:

  This PR is converting `test/lint/lint-all.sh` to `test/lint/lint-assertions.py`. It's an item of https://github.com/bitcoin/bitcoin/issues/24783.

ACKs for top commit:
  laanwj:
    Tested ACK 29f44fed36

Tree-SHA512: 5427936aaa8e009613048448cbd79d9225675bdcadcf4fbb70fd091e0aab85e350ef1678da1b388f70d62ca16f40409409f90da3f6bbf057480522cd50fde811
2022-04-28 12:28:30 +02:00
MacroFake fa60169811
rpc: Move signmessage RPC util to new file
Can be reviewed with --color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
2022-04-28 11:19:29 +02:00
MacroFake 9446de160f
Merge bitcoin/bitcoin#24831: tidy: add include-what-you-use
9b0a13a289 tidy: Add include-what-you-use (fanquake)
74cd038e30 refactor: fix includes in src/init (fanquake)
c79ad935f0 refactor: fix includes in src/compat (fanquake)

Pull request description:

  We recently added a [`clang-tidy` job](https://github.com/bitcoin/bitcoin/blob/master/ci/test/00_setup_env_native_tidy.sh) to the CI, which generates a compilation database. We can leverage that now existing database to begin running [include-what-you-use](https://include-what-you-use.org/) over the codebase.

  This PR demonstrates using a mapping_file to indicate fixups / includes that may differ from IWYU suggestions. In this case, I've added some fixups for glibc includes that I've [upstreamed changes for](https://github.com/include-what-you-use/include-what-you-use/pull/1026):
  ```bash
  # Fixups / upstreamed changes
  [
    { include: [ "<bits/termios-c_lflag.h>", private, "<termios.h>", public ] },
    { include: [ "<bits/termios-struct.h>", private, "<termios.h>", public ] },
    { include: [ "<bits/termios-tcflow.h>", private, "<termios.h>", public ] },
  ]
  ```

  The include "fixing" commits of this PR:
  * Adds missing includes.
  * Swaps C headers for their C++ counterparts.
  * Removes the pointless / unmaintainable `//for abc, xyz` comments. When using IWYU, if anyone wants to see / generate those comments, to see why something is included, it is trivial to do so (IWYU outputs them by default). i.e:
  ```cpp
  // The full include-list for compat/stdin.cpp:
  #include <compat/stdin.h>
  #include <poll.h>                  // for poll, pollfd, POLLIN
  #include <termios.h>               // for tcgetattr, tcsetattr
  #include <unistd.h>                // for isatty, STDIN_FILENO
  ```

  TODO:
  - [ ] Qt mapping_file. There is one in the IWYU repo, but it's for Qt 5.11. Needs testing.
  - [ ] Boost mapping_file. There is one in the IWYU repo, but it's for Boost 1.75. Needs testing.

  I'm not suggesting we turn this on the for entire codebase, or immediately go-nuts refactoring all includes. However I think our dependency includes are now slim enough, and our CI infrastructure in place such that we can start doing this in some capacity, and just automate away include fixups / refactorings etc.

ACKs for top commit:
  MarcoFalke:
    review ACK 9b0a13a289
  jonatack:
    ACK 9b0a13a289 reviewed changes and run CI output in https://cirrus-ci.com/task/4750910332076032

Tree-SHA512: 00beab5a5f2a6fc179abf08321a15391ecccaa91ab56f3c50c511e7b29a0d7c95d8bb43eac2c31489711086f6f77319d43d803cf8ea458e7cd234a780d9ae69e
2022-04-28 10:06:26 +02:00
Carl Dong 94ad45deb2 ci: Build libbitcoinkernel 2022-04-27 17:36:39 -04:00
fanquake 9c96f1008b
tidy: enable modernize-use-nullptr 2022-04-26 10:43:33 +01:00
hiago 29f44fed36 Converting `lint-all.sh` to `lint-all.py`.
Converting `lint-all.sh` to `lint-all.py`.
2022-04-26 06:25:01 -03:00
fanquake 9b0a13a289
tidy: Add include-what-you-use 2022-04-20 14:14:52 +01:00
Hennadii Stepanov a3a2bd9e8a
ci: Drop no longer needed package-specific flags 2022-04-16 08:59:13 +02:00
Hennadii Stepanov 86c2889518
ci: Make log verbose in error case only
This change silences depends build using LOG=1.
2022-04-14 13:44:29 +02:00
fanquake 864772c4f7
lint: mypy 0.942 2022-04-07 12:50:54 +01:00
fanquake 38031adee8
lint: flake8 4.0.1 2022-04-07 12:50:10 +01:00
fanquake eaf712c801
lint: codespell 2.1.0 2022-04-07 12:49:51 +01:00
MarcoFalke faa7ae8242
ci: Build all optional tools in tidy task 2022-04-06 12:48:38 +02:00
fanquake 372f1a3c25
Merge bitcoin/bitcoin#24753: ci: Add clang-tidy task
fab24f8c35 ci: Add clang-tidy task (MarcoFalke)

Pull request description:

  Fixes #24747

ACKs for top commit:
  fanquake:
    ACK fab24f8c35
  vincenzopalazzo:
    Code Review ACK fab24f8c35

Tree-SHA512: 87a5d67a23d540cea09925a6c186303c1249d4ca244b95940c5214860bf7a849fa12cc4fcf1bb9270b1004407124b3487902969d8ae9d2b14dae1fdfb57cc7e8
2022-04-06 10:10:41 +01:00
laanwj f421de5be6
Merge bitcoin/bitcoin#24236: Remove utxo db upgrade code
fa9112aac0 Remove utxo db upgrade code (MarcoFalke)

Pull request description:

  It is not possible to upgrade Bitcoin Core pre-segwit (pre-0.13.1) to a recent version without a full IBD from scratch after  commit 19a56d1519 (released in version 22.0).

  Any Bitcoin Core version with the new database format after commit 1088b02f0c (released in version 0.15), can upgrade to any version that is supported as of today.

  This leaves the versions 0.13.1-0.14.x. Even though those versions are unsupported, some users with an existing datadir may want to upgrade to a recent version. However, it seems reasonable to simply ask them to `-reindex` to run a full IBD from scratch. This allows us to remove the utxo db upgrade code.

ACKs for top commit:
  Sjors:
    re-ACK fa9112aac0
  laanwj:
    Code review ACK fa9112aac0

Tree-SHA512: 4243bb35df9ac4892f9fad30fe486d338745952bcff4160bcb0937c772d57b13b800647da14695e21e3655e85ee0d95fa3dc7789ee309d59ad84f422297fecb8
2022-04-05 15:38:14 +02:00
MarcoFalke fab24f8c35
ci: Add clang-tidy task 2022-04-04 11:57:06 +02:00
fanquake b0740fdcb8
ci: use DWARF-4 for Valgrind CI job
clang-14 defaults to using DWARF-5, which breaks vlagrinds (3.18) ability
to parse debug info. Valgrind claims to support DWARF-5 from version
3.18 onwards, but maybe that only works when building with GCC.

Explicitly use DWARF-4 for now. Note that from 11.0 GCC also defaults to
using DWARF-5.

https://releases.llvm.org/14.0.0/tools/clang/docs/ReleaseNotes.html#dwarf-support-in-clang
https://www.gnu.org/software/gcc/gcc-11/changes.html
https://valgrind.org/docs/manual/dist.news.html
2022-04-04 10:39:55 +01:00
fanquake 74b011bbfa
Merge bitcoin/bitcoin#23008: ci: Use clang-12 and libcxx-12 for msan
fa73f8a469 ci: Use clang-12 and libcxx-12 for msan (MarcoFalke)

Pull request description:

  Run the latest sanitizers to get the most implemented features

ACKs for top commit:
  fanquake:
    ACK fa73f8a469 - `--disable-hardening` matches what was just added to oss-fuzz.

Tree-SHA512: 2e533bb9273c97600176be2e41069a03f425aa586f9f32b8ed5f0c9844215a3a41e95a8edd58d044386e350807d6a1df09008a7da35428abd185a509ca71bd82
2022-03-30 19:41:16 +01:00
MarcoFalke fa73f8a469
ci: Use clang-12 and libcxx-12 for msan 2022-03-30 11:56:06 +02:00
laanwj bdbabc50ba
Merge bitcoin/bitcoin#24561: guix: use LIEF 0.12.0
3c3bd90220 test: remove lief install from multiprocess job (fanquake)
983e0a2058 contrib: use LIEF 0.12.0 for symbol and security checks (fanquake)

Pull request description:

  [LIEF](https://github.com/lief-project/LIEF) `0.12.0` is [now available](https://github.com/lief-project/LIEF/releases/tag/0.12.0), and includes minor changes we have upstreamed to improve support for RISC-V binaries:
  * https://github.com/lief-project/LIEF/pull/562
  * https://github.com/lief-project/LIEF/pull/640

  Closes: #23655.

  Guix Build (x86_64):
  ```bash
  2fc504baf4f550092f05115f8c7c9046ec2988f097b55b33c4c963e02f11cfc3  guix-build-3c3bd9022026/output/aarch64-linux-gnu/SHA256SUMS.part
  15b1a8a33f4b750cf8d3e2753e6fe3ca7560aeefa88c1acbaf0be412cbd81cc4  guix-build-3c3bd9022026/output/aarch64-linux-gnu/bitcoin-3c3bd9022026-aarch64-linux-gnu-debug.tar.gz
  77a446792dba101afd565f0b4441c5dfe8e08bf9e4c9ea6e862a0df118a55327  guix-build-3c3bd9022026/output/aarch64-linux-gnu/bitcoin-3c3bd9022026-aarch64-linux-gnu.tar.gz
  dc63f742a3d396ab2b2115f7cea4e90c127267a0c9fc2f1e9af9e052a670cd41  guix-build-3c3bd9022026/output/arm-linux-gnueabihf/SHA256SUMS.part
  5214a5c3e532ddfec218354b6512cc4eedd6411a199cc2d4eed7d0ad77873a0e  guix-build-3c3bd9022026/output/arm-linux-gnueabihf/bitcoin-3c3bd9022026-arm-linux-gnueabihf-debug.tar.gz
  fea7b00ca6d8b7bc72c2e4e2c9e9aa94c76f9f463af82cf0bd5a6fc8951a7fe7  guix-build-3c3bd9022026/output/arm-linux-gnueabihf/bitcoin-3c3bd9022026-arm-linux-gnueabihf.tar.gz
  200bf59ecaebdcb2a1302160dd47288135b8cead99e202d1a8b7da8d54a872c3  guix-build-3c3bd9022026/output/arm64-apple-darwin/SHA256SUMS.part
  51c079c29966a1f76f690f28ed99d767413327422636bbbe9ca499e97954cfd4  guix-build-3c3bd9022026/output/arm64-apple-darwin/bitcoin-3c3bd9022026-arm64-apple-darwin-unsigned.dmg
  2e85d002b7e993a90b23363bb01d8dca644fbabd9debb336012834abeddce024  guix-build-3c3bd9022026/output/arm64-apple-darwin/bitcoin-3c3bd9022026-arm64-apple-darwin-unsigned.tar.gz
  257e272f6ed867f7634d1bba5860da855e35793646581b37518e25d1fdab48db  guix-build-3c3bd9022026/output/arm64-apple-darwin/bitcoin-3c3bd9022026-arm64-apple-darwin.tar.gz
  a0c755bb31763995e1d2050d5448bb98194c238fdd6b9eff901e429c9fdfa0f6  guix-build-3c3bd9022026/output/dist-archive/bitcoin-3c3bd9022026.tar.gz
  cf07cfefe010a1cc58ceab5dd5361ca39d770038d73682bd60b52410708c5f35  guix-build-3c3bd9022026/output/powerpc64-linux-gnu/SHA256SUMS.part
  aa93521bfbf7cf693fe56500faa6ac460bb297e5af2d8cf4e857f7ea0e92eb34  guix-build-3c3bd9022026/output/powerpc64-linux-gnu/bitcoin-3c3bd9022026-powerpc64-linux-gnu-debug.tar.gz
  883c2e5fe0d8215dcc0a252cd62c8fd95fc13b31e1bd72d340dfb2e4adadfe01  guix-build-3c3bd9022026/output/powerpc64-linux-gnu/bitcoin-3c3bd9022026-powerpc64-linux-gnu.tar.gz
  903666488f620a5710bc1709f52feb60d530aaba6e7e0b200692fa2945e569b8  guix-build-3c3bd9022026/output/powerpc64le-linux-gnu/SHA256SUMS.part
  2dd8a3fa924111e839097da8ee12675a265e64aeeae270f1f3e86f9c9b7bd28a  guix-build-3c3bd9022026/output/powerpc64le-linux-gnu/bitcoin-3c3bd9022026-powerpc64le-linux-gnu-debug.tar.gz
  d93a4347ea376e02760c52bb836317fd3f9c8e78ff004d9c141ebce185f3d364  guix-build-3c3bd9022026/output/powerpc64le-linux-gnu/bitcoin-3c3bd9022026-powerpc64le-linux-gnu.tar.gz
  7ee00ec4a2cd83f27d619c6983e93baa198d78167a860e8d9f4c4aa670120ab2  guix-build-3c3bd9022026/output/riscv64-linux-gnu/SHA256SUMS.part
  9ad79a47c59a67f881ba2ef7a669ffb667b6e0bd13407730574dc7af542fd935  guix-build-3c3bd9022026/output/riscv64-linux-gnu/bitcoin-3c3bd9022026-riscv64-linux-gnu-debug.tar.gz
  35462865447683571b006d426e532090c19ef18b9f399b84ebf37795fb3c4eaa  guix-build-3c3bd9022026/output/riscv64-linux-gnu/bitcoin-3c3bd9022026-riscv64-linux-gnu.tar.gz
  a5a3bb0527ded1eea3956208d1dc7289e39786594ac4cbee17f4c28573b7e4fb  guix-build-3c3bd9022026/output/x86_64-apple-darwin/SHA256SUMS.part
  e75435885e97dec09f3d9c51c0bd5987329a485ef26027d6e7b4cafcec20a9ab  guix-build-3c3bd9022026/output/x86_64-apple-darwin/bitcoin-3c3bd9022026-x86_64-apple-darwin-unsigned.dmg
  438b027186095eed1615898a403b0bdd0a686d95f0617b4065db469c8c4e732c  guix-build-3c3bd9022026/output/x86_64-apple-darwin/bitcoin-3c3bd9022026-x86_64-apple-darwin-unsigned.tar.gz
  1d33e0661041f00b4f224f231bbb71bda534ad8ad76a7d2b041084911cdd0814  guix-build-3c3bd9022026/output/x86_64-apple-darwin/bitcoin-3c3bd9022026-x86_64-apple-darwin.tar.gz
  fc9a88c1cdf10570bc2c453c1a643a04b525155f0b98a18c90b8d5fd64fcf630  guix-build-3c3bd9022026/output/x86_64-linux-gnu/SHA256SUMS.part
  435896b6a85f46a2f3f0b5f278936e84d0212ef7102787b70f139648373344d3  guix-build-3c3bd9022026/output/x86_64-linux-gnu/bitcoin-3c3bd9022026-x86_64-linux-gnu-debug.tar.gz
  02c84c4c74846740b5336e8889194909bb6d638cf731851ddd8af7bbffd1ae9b  guix-build-3c3bd9022026/output/x86_64-linux-gnu/bitcoin-3c3bd9022026-x86_64-linux-gnu.tar.gz
  d4176cc40eaa0769410717b28637d507ddef182c73c60801f333ddd7a56b41f7  guix-build-3c3bd9022026/output/x86_64-w64-mingw32/SHA256SUMS.part
  7623a149798beba635c21639975d5cbf2a417c7e8a1bf3583abe1bb20cfa1673  guix-build-3c3bd9022026/output/x86_64-w64-mingw32/bitcoin-3c3bd9022026-win64-debug.zip
  a1de00c9d4315928e3ce93890009fe2ea8f3f1fa7ca63004d952192012131d20  guix-build-3c3bd9022026/output/x86_64-w64-mingw32/bitcoin-3c3bd9022026-win64-setup-unsigned.exe
  82dc001a7f6d2b405b89bd343276db59954f25461d828ddf2e3c42f5ba4fe164  guix-build-3c3bd9022026/output/x86_64-w64-mingw32/bitcoin-3c3bd9022026-win64-unsigned.tar.gz
  6fc6f6101672a4cee2696a1517def03b6872435d968d74a47efbb4f8c5198c03  guix-build-3c3bd9022026/output/x86_64-w64-mingw32/bitcoin-3c3bd9022026-win64.zip
  ```

  Guix Build (arm64):
  ```bash
  d803406d4a90f9813bff58f01fde109d77dbf5b37b1af68e7a58fb4c3041a390  guix-build-3c3bd9022026/output/arm-linux-gnueabihf/SHA256SUMS.part
  25a234f4c0eb0ed280d7c24b65e2bae75d44f4fade1895d9d6d2a3880ed29557  guix-build-3c3bd9022026/output/arm-linux-gnueabihf/bitcoin-3c3bd9022026-arm-linux-gnueabihf-debug.tar.gz
  0b6044ea0441536e7b2853cce5a354fff33625db8c18c70023c78a9de5364376  guix-build-3c3bd9022026/output/arm-linux-gnueabihf/bitcoin-3c3bd9022026-arm-linux-gnueabihf.tar.gz
  c44810505ee8f7800488c0442edc71550a1f989598c702e9a31709b9880ef8ba  guix-build-3c3bd9022026/output/arm64-apple-darwin/SHA256SUMS.part
  ce9a74deac4a20a2ca4d7f8dfc242aea367c2aa55cb3b9de4e11c691ce0755bd  guix-build-3c3bd9022026/output/arm64-apple-darwin/bitcoin-3c3bd9022026-arm64-apple-darwin-unsigned.dmg
  341efb7b536ad6f75456dd51134e1afa4773708be2103733c48c8fb7a85df8b0  guix-build-3c3bd9022026/output/arm64-apple-darwin/bitcoin-3c3bd9022026-arm64-apple-darwin-unsigned.tar.gz
  607151e3423f712ea1bb0e24b7f625290e2decfe4586adf193e3218f62e87f71  guix-build-3c3bd9022026/output/arm64-apple-darwin/bitcoin-3c3bd9022026-arm64-apple-darwin.tar.gz
  a0c755bb31763995e1d2050d5448bb98194c238fdd6b9eff901e429c9fdfa0f6  guix-build-3c3bd9022026/output/dist-archive/bitcoin-3c3bd9022026.tar.gz
  16f1f669f9f7a658ad25d72e628c68fefdc3469857efd06149b0ae063ad07cf2  guix-build-3c3bd9022026/output/powerpc64-linux-gnu/SHA256SUMS.part
  8c36e24b12d0f3c200b51c813a6c8e85b826c994af4891a4eed16bb2447f6ae9  guix-build-3c3bd9022026/output/powerpc64-linux-gnu/bitcoin-3c3bd9022026-powerpc64-linux-gnu-debug.tar.gz
  89777d8c44ec0ce3e59e74461fb7ed39b544ff1d40ed8399e3a26733fcd938af  guix-build-3c3bd9022026/output/powerpc64-linux-gnu/bitcoin-3c3bd9022026-powerpc64-linux-gnu.tar.gz
  f51279dc699a1b8625f8fd3e13ec1555e81be7696dc0fd88621177fb4c3353d7  guix-build-3c3bd9022026/output/powerpc64le-linux-gnu/SHA256SUMS.part
  15c1aad2b67838fbf004cfd4a0926cb2e2f2d0fcee639e5cf17fec9c659be7ad  guix-build-3c3bd9022026/output/powerpc64le-linux-gnu/bitcoin-3c3bd9022026-powerpc64le-linux-gnu-debug.tar.gz
  102ae28c667fee645fb1541395531742b89886e153ed979639532c8cab50d24b  guix-build-3c3bd9022026/output/powerpc64le-linux-gnu/bitcoin-3c3bd9022026-powerpc64le-linux-gnu.tar.gz
  ccff48fd8b5a26d32d2743f430d812a449ad00ee502fc1f02538598ff7def9db  guix-build-3c3bd9022026/output/riscv64-linux-gnu/SHA256SUMS.part
  d038064cfb33dee85833a58ee5a7c40c94041aa0574da9fa14837b0da7a92b3a  guix-build-3c3bd9022026/output/riscv64-linux-gnu/bitcoin-3c3bd9022026-riscv64-linux-gnu-debug.tar.gz
  763a0c5c7492d11370a0208f3110fc73f6bac3526b401d1df3eca5cff1037a36  guix-build-3c3bd9022026/output/riscv64-linux-gnu/bitcoin-3c3bd9022026-riscv64-linux-gnu.tar.gz
  14aad526acfed07e5781efa8febe37f002c5772090aad0cd98cf7f4cf0cc3177  guix-build-3c3bd9022026/output/x86_64-apple-darwin/SHA256SUMS.part
  e75435885e97dec09f3d9c51c0bd5987329a485ef26027d6e7b4cafcec20a9ab  guix-build-3c3bd9022026/output/x86_64-apple-darwin/bitcoin-3c3bd9022026-x86_64-apple-darwin-unsigned.dmg
  742cacbb471968c396da97bbed0332220b721c552b663e931b9bfeb0e4b2f4de  guix-build-3c3bd9022026/output/x86_64-apple-darwin/bitcoin-3c3bd9022026-x86_64-apple-darwin-unsigned.tar.gz
  1d33e0661041f00b4f224f231bbb71bda534ad8ad76a7d2b041084911cdd0814  guix-build-3c3bd9022026/output/x86_64-apple-darwin/bitcoin-3c3bd9022026-x86_64-apple-darwin.tar.gz
  b8fdd26c3aef78d13161f6119d755a853470ced695cb40dcc8ff9f0463a533e1  guix-build-3c3bd9022026/output/x86_64-linux-gnu/SHA256SUMS.part
  48846704f00e92e1cb6898a13e077da5a314314d6aa694b022d0f2aa68b7128b  guix-build-3c3bd9022026/output/x86_64-linux-gnu/bitcoin-3c3bd9022026-x86_64-linux-gnu-debug.tar.gz
  52968c1aea392d19989613b4faf0075702c9aa834f74cd3e7a09883e201ae3e4  guix-build-3c3bd9022026/output/x86_64-linux-gnu/bitcoin-3c3bd9022026-x86_64-linux-gnu.tar.gz
  a4df21e146042c6d78b0724ace37fdb05ce3a4ce8c1ed8274727b031cf53805e  guix-build-3c3bd9022026/output/x86_64-w64-mingw32/SHA256SUMS.part
  68f05fc5cc5e157805280bcb36fdbc6873db9e7b3afb4ddd992786db722530fc  guix-build-3c3bd9022026/output/x86_64-w64-mingw32/bitcoin-3c3bd9022026-win64-debug.zip
  a1de00c9d4315928e3ce93890009fe2ea8f3f1fa7ca63004d952192012131d20  guix-build-3c3bd9022026/output/x86_64-w64-mingw32/bitcoin-3c3bd9022026-win64-setup-unsigned.exe
  82dc001a7f6d2b405b89bd343276db59954f25461d828ddf2e3c42f5ba4fe164  guix-build-3c3bd9022026/output/x86_64-w64-mingw32/bitcoin-3c3bd9022026-win64-unsigned.tar.gz
  b644760f08a6147bccae8761e33fa13dc7f0cf391240521a81e57b437e272baa  guix-build-3c3bd9022026/output/x86_64-w64-mingw32/bitcoin-3c3bd9022026-win64.zip
  ```

ACKs for top commit:
  laanwj:
    Code review ACK 3c3bd90220
  hebasto:
    ACK 3c3bd90220, tested that bitcoin/bitcoin#23655 has been resolved.

Tree-SHA512: abc95ab68d3973c89d421ee7a5b795f6fa802dc665db47529d5f9aee5b92b8a7b55f9a45c634b9be6e917038e67e785f1809cc189c84be13f089f4e7fa1029f9
2022-03-30 11:10:25 +02:00
fanquake 896beca831
Merge bitcoin/bitcoin#24708: ci: Avoid boost test warnings
fa76b2fbe3 ci: Avoid boost test warnings (MarcoFalke)

Pull request description:

  This removes boost test warnings from the CI log.

  For example https://cirrus-ci.com/task/6029158399606784?logs=ci#L4060:

  ```
  Boost.Test WARNING: token "/tmp/cirrus-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/test_bitcoin-addrman_tests.o" does not correspond to the Boost.Test argument
                      and should be placed after all Boost.Test arguments and the -- separator.
                      For example: test_bitcoin --random -- /tmp/cirrus-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/test_bitcoin-addrman_tests.o
  Boost.Test WARNING: token "/tmp/cirrus-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/test_bitcoin-allocator_tests.o" does not correspond to the Boost.Test argument
                      and should be placed after all Boost.Test arguments and the -- separator.
                      For example: test_bitcoin --random -- /tmp/cirrus-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/test_bitcoin-allocator_tests.o
  Boost.Test WARNING: token "/tmp/cirrus-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/test_bitcoin-amount_tests.o" does not correspond to the Boost.Test argument
                      and should be placed after all Boost.Test arguments and the -- separator.
  ...

ACKs for top commit:
  fanquake:
    ACK fa76b2fbe3 - checked that tests are also still being run in other jobs.

Tree-SHA512: c00fe1242ee6b9ef92d511f1e86305c1731d8894bb89ec8fcdf30069959831483933a93a6bfd33d9f2d15706bdeedbba29e82c409eed266abfc57aabfdaf4841
2022-03-30 08:54:26 +01:00
MarcoFalke ecf692b466
Merge bitcoin/bitcoin#24540: ci: Integrate ccache into MSVC build
3a53927f03 ci: Integrate ccache into MSVC build (Hennadii Stepanov)

Pull request description:

  [ccache 4.6](https://ccache.dev/releasenotes.html#_ccache_4_6):
  > Added support for caching calls to Microsoft Visual C++ (MSVC)

  Integrated into our native Windows CI task.

  [On master](url) (c109e7d51c):
  ![Screenshot from 2022-03-12 10-17-58](https://user-images.githubusercontent.com/32963518/158012098-7ac9d441-2eb0-481e-bcc5-3700c1ce2b15.png)

  [This PR](https://cirrus-ci.com/task/6572984340054016):
  ![Screenshot from 2022-03-12 10-25-33](https://user-images.githubusercontent.com/32963518/158012361-d6bf88bc-f98d-4771-8b4f-31bf5673d085.png)

  ```
  Summary:
    Hits:             222 /  222 (100.0 %)
      Direct:         222 /  222 (100.0 %)
      Preprocessed:     0 /    0
    Misses:             0
      Direct:           0
      Preprocessed:     0
    Errors:             7
    Uncacheable:        9
  Primary storage:
    Hits:             444 /  444 (100.0 %)
    Misses:             0
    Cache size (GB): 0.04 / 5.00 (0.86 %)

  Use the -v/--verbose option for more details.
  ```

Top commit has no ACKs.

Tree-SHA512: d8cf91d8e75a5187cf456960bdf759f857fb9a9b0c4087e5e46ccbe2202aca5f1b9b38a47ec54d98e885c0f5d78de93a3188fb966fa5b346e81907c211ba1e79
2022-03-29 20:22:48 +02:00
MarcoFalke fa76b2fbe3
ci: Avoid boost test warnings 2022-03-29 16:05:39 +02:00
fanquake 527eeaf580
ci: note why BDB is disabled for MSAN jobs 2022-03-29 10:59:43 +01:00
fanquake d6c71b0ccf
ci: remove explicit --enable-wallet from msan job
This isn't required to enable sqlite wallets, as support is
automatically detected when sqlite is available.
2022-03-29 10:33:57 +01:00
fanquake 3c3bd90220
test: remove lief install from multiprocess job
This job doesn't run any security / symbol checks, so lief isn't needed.
2022-03-28 10:35:37 +01:00
MarcoFalke fa6e47d85b
ci: Compile fuzz binary in periodic task 2022-03-25 10:17:14 +01:00
MarcoFalke 999982b06c
build: Add --enable-c++20 option 2022-03-24 11:37:42 +01:00
MarcoFalke fa43933e3b
ci: Temporarily use clang-13 to work around clang-14 TSan bug 2022-03-15 18:24:16 +01:00
MarcoFalke e7db4e245a
Merge bitcoin/bitcoin#24164: build: Bump minimum required clang/libc++ to 8.0
fae20e6b50 Revert "Avoid the use of P0083R3 std::set::merge" (MarcoFalke)
fab53b5fd4 ci/doc: Set minimum required clang/libc++ version to 8.0 (MarcoFalke)

Pull request description:

  This is not for 23.0, but for 24.0. It comes with the following benefits:

  * Can use C++17 P0083R3 std::set::merge from libc++ 8.0
  * No longer need to provide support for clang-7, which already fails to compile on some architectures (https://github.com/bitcoin/bitcoin/issues/21294#issuecomment-998098483)

  This should be fine, given that all supported operating systems ship with at least clang-10:

  * CentOS 8: clang-12
  * Stretch: https://packages.debian.org/stretch/clang-11
  * Buster: https://packages.debian.org/buster-backports/clang-11
  * Bionic: https://packages.ubuntu.com/bionic-updates/clang-10
  * Focal: https://packages.ubuntu.com/focal/clang-10

ACKs for top commit:
  fanquake:
    ACK fae20e6b50 - I think this is fine to do. I would be surprised if in another 6 months time someone was stuck on a system we supported, needing to compile Core, and only had access to Clang 7 or older. As mentioned in the PR description, all systems we currently support, already support multiple newer versions of Clang.
  hebasto:
    ACK fae20e6b50, I have reviewed the code and it looks OK, I agree it can be merged.

Tree-SHA512: 3b4c6c130ff40dd7e84934af076863415e5dd661d823c72e3e3832566c65be6e877a7ef9164bbcf394bcea4b897fc29a48db0f231c22ace0e2c9b5638659a628
2022-03-12 10:37:05 +01:00
Hennadii Stepanov 3a53927f03
ci: Integrate ccache into MSVC build 2022-03-12 03:20:33 +01:00
fanquake 3566353c5e
ci: remove compiled-but-unused BDB from MSAN job
Self-compiled BDB was added to this job as opposed to using depends BDB
due to linking issues, however the compiled BDB is not actually used.
Remove it for now, given we don't actually lose any coverage (note that
BDB is also no used the MSAN fuzz job), and in future, we can use
depends BDB.
2022-03-10 12:48:21 +00:00
MarcoFalke fa9112aac0
Remove utxo db upgrade code 2022-03-10 13:05:29 +01:00
fanquake c9ed9927bb
Merge bitcoin/bitcoin#24132: build: Bump minimum Qt version to 5.11.3
956f7322f6 build: Bump minimum Qt version to 5.11.3 (Hennadii Stepanov)
e22d10b936 ci: Switch from bionic to buster (Hennadii Stepanov)

Pull request description:

  The current minimum Qt version is 5.9.5 which has been set in bitcoin/bitcoin#21286.

  Distro support:
  - centos 7 -- unsupported since bitcoin/bitcoin#23511
  - centos 8 -- [5.15.2](http://mirror.centos.org/centos/8/AppStream/x86_64/os/Packages/qt5-qtbase-5.15.2-3.el8.x86_64.rpm)
  - buster -- [5.11.3](https://packages.debian.org/buster/libqt5core5a)
  - bullseye  -- [5.15.2](https://packages.debian.org/bullseye/libqt5core5a)
  - _bionic_ -- [5.9.5](https://packages.ubuntu.com/bionic/libqt5core5a)
  - focal -- [5.12.8](https://packages.ubuntu.com/focal/libqt5core5a)

  As another Ubuntu LTS is coming soon, it seems unreasonable to stick to Qt 5.9 which support [ended](https://www.qt.io/blog/2017/06/07/renewed-qt-support-services) on 2020-05-31. Anyway, it's still possible to build Bitcoin Core GUI with depends on bionic system.

  Bumping the minimum Qt version allows to make code safer and more reliable, e.g.:
  - functor-parameter overload of [`QMetaObject::invokeMethod`](https://doc.qt.io/qt-5/qmetaobject.html#invokeMethod-4)
  - fixed https://bugreports.qt.io/browse/QTBUG-10907

  An example of the patch using the functor-overload of `QMetaObject::invokeMethod`:
  ```diff
  --- a/src/qt/walletmodel.cpp
  +++ b/src/qt/walletmodel.cpp
  @@ -349,7 +349,7 @@ bool WalletModel::changePassphrase(const SecureString &oldPass, const SecureStri
   static void NotifyUnload(WalletModel* walletModel)
   {
       qDebug() << "NotifyUnload";
  -    bool invoked = QMetaObject::invokeMethod(walletModel, "unload");
  +    bool invoked = QMetaObject::invokeMethod(walletModel, &WalletModel::unload);
       assert(invoked);
   }

  ```
  It uses the same new syntax as signal-slot connection with compile-time check. Also see bitcoin/bitcoin#16348.

  This PR is intended to be merged early [after](https://github.com/bitcoin/bitcoin/issues/22969) branching `23.x` off.

ACKs for top commit:
  MarcoFalke:
    cr ACK 956f7322f6
  fanquake:
    ACK 956f7322f6

Tree-SHA512: 3d652bcdcd990ce785ad412ed70234d4f27743895e535a53ed44b35d4afc3052e066c4c84f417e30bc53d0a3dd9ebed62444c57b7c765cb1e9aa687fbf866877
2022-03-07 14:53:23 +00:00
MarcoFalke fab53b5fd4
ci/doc: Set minimum required clang/libc++ version to 8.0 2022-03-04 14:13:21 +00:00
MarcoFalke 619f8a27ad
Merge bitcoin/bitcoin#24304: [kernel 0/n] Introduce `bitcoin-chainstate`
2c03cec2ff ci: Build bitcoin-chainstate (Carl Dong)
095aa6ca37 build: Add example bitcoin-chainstate executable (Carl Dong)

Pull request description:

  Part of: #24303

  This PR introduces an example/demo `bitcoin-chainstate` executable using said library which can print out information about a datadir and take in new blocks on stdin.

  Please read the commit messages for more details.

  -----

  #### You may ask: WTF?! Why is `index/*.cpp`, etc. being linked in?

  This PR is meant only to capture the state of dependencies in our consensus engine as of right now. There are many things to decouple from consensus, which will be done in subsequent PRs. Listing the files out right now in `bitcoin_chainstate_SOURCES` is purely to give us a clear picture of the task at hand, it is **not** to say that these dependencies _belongs_ there in any way.

  ### TODO

  1. Clean up `bitcoin-chainstate.cpp`
     It is quite ugly, with a lot of comments I've left for myself, I should clean it up to the best of my abilities (the ugliness of our init/shutdown might be the upper bound on cleanliness here...)

ACKs for top commit:
  ajtowns:
    ACK 2c03cec2ff
  ryanofsky:
    Code review ACK 2c03cec2ff. Just rebase, comments, formatting change since last review
  MarcoFalke:
    re-ACK 2c03cec2ff 🏔

Tree-SHA512: 86e7fb5718caa577df8abc8288c754f4a590650d974df9d2f6476c87ed25c70f923c4db651c6963f33498fc7a3a31f6692b9a75cbc996bf4888c5dac2f34a13b
2022-03-03 19:31:36 +00:00
laanwj 25290071c4
Merge bitcoin/bitcoin#24397: build: Fix Boost.Process check for Boost 1.73 and older
774323e378 ci: Force `--enable-external-signer` to prevent future regressions (Hennadii Stepanov)
69978858a4 build: Fix Boost.Process check for Boost 1.73 and older (Hennadii Stepanov)
2199ef79cb build: Fix a non-portable use of `test` (Hennadii Stepanov)
d436c488d4 build, refactor: Replace tabs with spaces (Hennadii Stepanov)

Pull request description:

  On master (5f44c5c428) Boost.Process check false fails without the `-lpthread` flag.

  ```
  $ grep -C 2 pthread_detach config.log
  /usr/bin/ld: /tmp/cczCQfQv.o: in function `boost::asio::detail::posix_global_impl<boost::asio::system_context>::~posix_global_impl()':
  conftest.cpp:(.text._ZN5boost4asio6detail17posix_global_implINS0_14system_contextEED2Ev[_ZN5boost4asio6detail17posix_global_implINS0_14system_contextEED5Ev]+0xa3): undefined reference to `pthread_join'
  /usr/bin/ld: conftest.cpp:(.text._ZN5boost4asio6detail17posix_global_implINS0_14system_contextEED2Ev[_ZN5boost4asio6detail17posix_global_implINS0_14system_contextEED5Ev]+0xc4): undefined reference to `pthread_detach'
  collect2: error: ld returned 1 exit status
  configure:26674: $? = 1
  ```

  Not required for Boost 1.74+.

ACKs for top commit:
  laanwj:
    Code review ACK 774323e378, is a bugfix/workaround, seems fine to merge last minute for 23.0.

Tree-SHA512: 2a9d4b67fd8910e107af972d8c223286b7c933bc310616f86c8b6d8c903438916980fc76bd7e37f2698f6ce5361dc706cbf2933d1ac2c081bcabe1b83ca7d6b6
2022-02-28 14:06:16 +01:00
MarcoFalke f1ce67f09f
Merge bitcoin/bitcoin#19013: test: add v0.20.1, v0.21.0 and v22.0 to backwards compatibility test
24cec4b5c0 test: Fix intermittent test failure in feature_backwards_compatibility (MarcoFalke)
d8b705f1ca test: previous releases: add v22.0 (Sjors Provoost)
40849eebd9 test: bump sandbox argument minimum version (Sjors Provoost)
8a57a06a50 test: previous releases: add v0.21.0 (Sjors Provoost)
8cba75f5fd test: v0.20.1 backwards compatibility (Sjors Provoost)
0e4b695b6a test: backwards compatibility: misc fixes (Sjors Provoost)
76557cbe4c test: Remove i686 from test/get_previous_releases.py (MarcoFalke)

Pull request description:

  This also simplifies the tests a bit.

ACKs for top commit:
  ryanofsky:
    Code review ACK 24cec4b5c0. Only change since last review is rebasing and adding comment and whitelist args.

Tree-SHA512: 85a603ddd70fd8f0180d00fb84eb2ad2f92d6199b7d3f7c1abd660bfba53f869faf40f1a4183a8ce15dbd496ee3132d879c1258651c9d443ece69e5fe328bd26
2022-02-24 17:42:28 +01:00
Hennadii Stepanov 774323e378
ci: Force `--enable-external-signer` to prevent future regressions 2022-02-22 14:27:30 +02:00
fanquake 2ab4fbe375
Merge bitcoin/bitcoin#24292: Revert "ci: Run fuzzer task for the master branch only"
fa27745ccb ci: Bump fuzz tasks to jammy (MarcoFalke)
fab8cd5f87 Revert "ci: Run fuzzer task for the master branch only" (MarcoFalke)

Pull request description:

  This reverts commit 5a9e255e5a.

  I think we should attempt to maintain the fuzz tasks for release branches as well.

  If it is too difficult for one branch, it could make sense to disable it for that branch, but not for all branches unconditionally.

  Also, bump to jammy.

ACKs for top commit:
  fanquake:
    ACK fa27745ccb - we'll see how we go with the 23.x release branch.

Tree-SHA512: d6d08e7dce0884b556c51ff1896aebbbb5a805c22decd58af81a04192d19876978696017b489ec55886ddfd5c022963baaab5f11022369ae5291016826ff8017
2022-02-21 13:12:36 +00:00
MarcoFalke 3de5fcc94f
Merge bitcoin/bitcoin#24296: ci: use Ubuntu Jammy for Windows cross-compilation CI
a1515cdd96 ci: use Ubuntu Jammy for Windows CI (fanquake)

Pull request description:

  This means we'll compile using [GCC 10.3.x](https://packages.ubuntu.com/jammy/g++-mingw-w64) and [mingw-w64 8.0.0](https://packages.ubuntu.com/jammy/mingw-w64) which better matches our Guix release environment.

ACKs for top commit:
  MarcoFalke:
    cr ACK a1515cdd96
  hebasto:
    ACK a1515cdd96, I have reviewed the code and it looks OK, I agree it can be merged.

Tree-SHA512: a57cce1874324c9dd00e5d8989996d214facbdd561440471c15e6cc1808bca1c6fd758abe7a1b87378b2e7f9c25e7c9d8242df911cd1ef6cfbe49718adc3be5d
2022-02-21 07:55:36 +01:00
fanquake a1515cdd96
ci: use Ubuntu Jammy for Windows CI
This means we'll compile using GCC 10.3.x and mingw-w64 8.0.0 which
better matches our Guix release environment.
2022-02-18 15:19:05 +00:00
fanquake 7e02adf78d
ci: add missing sqlite_cflags to MSAN fuzz job
These are present in the other MSAN job.
2022-02-18 14:10:59 +00:00
fanquake 54d817c5b9
ci: remove boost_cxxflags from MSAN CIs
No-longer needed after #24301.
2022-02-18 14:07:55 +00:00
Carl Dong 2c03cec2ff ci: Build bitcoin-chainstate
...to make sure that the linker errors that arise from coupling
regressions are caught by CI.

Adding to the "no wallet" ci job as suggested by MarcoFalke.
2022-02-14 14:54:01 -05:00
fanquake 39e66e938f
build: use header-only Boost unit test 2022-02-13 20:59:02 +00:00
MarcoFalke fa27745ccb
ci: Bump fuzz tasks to jammy
This gives them a newer clang version, which may have more sanitizers
available.
2022-02-08 20:23:21 +01:00
Hennadii Stepanov e22d10b936
ci: Switch from bionic to buster
This change is a prerequisite for the following bumping Qt minimum
version to 5.11.3. It is required as bionic has Qt 5.9.5.

Effectively, this also changes:
- gcc from 8.4.0 to 8.3.0
- python from 3.6.5 to 3.7.3
2022-02-05 23:53:37 +02:00
MarcoFalke fa4b61911d
test: Remove unused valgrind suppressions 2022-02-04 16:35:15 +01:00
MarcoFalke faccb2d7fe
test: Exclude broken feature_init for now 2022-02-04 16:33:36 +01:00
fanquake 07269321f3
build: remove Boost::system usage 2022-02-03 18:35:52 +08:00
Kiminuo b87f9c5edf
build: remove boost::filesystem usage 2022-02-03 18:35:52 +08:00
Hennadii Stepanov 0bbae237a8
ci: Drop no longer needed `update-alternatives` 2022-02-02 19:29:01 +02:00
MarcoFalke fafc55a489
ci: Use dash when building depends in centos build 2022-02-01 10:15:17 +01:00
MarcoFalke fa33236e77
scripted-diff: Rename DOCKER_EXEC to CI_EXEC
-BEGIN VERIFY SCRIPT-
 sed -i "s/DOCKER_EXEC/CI_EXEC/g" $(git grep -l DOCKER_EXEC)
-END VERIFY SCRIPT-
2022-02-01 10:14:53 +01:00
MarcoFalke fa5457e64a
ci: Bump CentOS 8 image 2022-02-01 10:13:25 +01:00
fanquake 446e73cc0b
build: use macOS 11 SDK (Xcode 12.2)
This should be sufficient to support building for Apple ARM when
cross-compiling.
2022-01-26 17:28:16 +08:00
fanquake e2ab9f83f8
build: disable external signer on Windows 2022-01-15 10:02:04 +08:00
MarcoFalke 95833c012e
Merge bitcoin/bitcoin#23938: ci: Clone entire bitcoin-core/qa-assets repo only when run fuzzing
0b7c55f157 ci: Clone entire bitcoin-core/qa-assets repo only when run fuzzing (Hennadii Stepanov)

Pull request description:

  This PR speeds up CI tasks that run unit tests but do not run fuzzing.

  On my machine:
  ```
  $ time git clone --depth=1 https://github.com/bitcoin-core/qa-assets
  Cloning into 'qa-assets'...
  remote: Enumerating objects: 289750, done.
  remote: Counting objects: 100% (289750/289750), done.
  remote: Compressing objects: 100% (207687/207687), done.
  remote: Total 289750 (delta 16863), reused 275449 (delta 12092), pack-reused 0
  Receiving objects: 100% (289750/289750), 1.39 GiB | 4.79 MiB/s, done.
  Resolving deltas: 100% (16863/16863), done.
  Updating files: 100% (294515/294515), done.

  real7m43,417s
  user2m39,771s
  sys0m43,272s
  ```

ACKs for top commit:
  MarcoFalke:
    cr ACK 0b7c55f157

Tree-SHA512: 8f8dd63e361c3d9c64a2b31f207f59beea7d2e4142363f7762ec724a61152323fda0c6b98a36386e15f2ca4ef3f7412da763baef65e862fc7f48cdeda8c40e09
2022-01-06 17:15:17 +01:00
Hennadii Stepanov 0b7c55f157
ci: Clone entire bitcoin-core/qa-assets repo only when run fuzzing 2022-01-06 16:23:48 +02:00
fanquake e09773d20a
build: use a static .tiff for macOS .dmg over generating
Co-authored-by: Pavol Rusnak <pavol@rusnak.io>
2022-01-02 15:38:19 +08:00
MarcoFalke 623745ca74
Merge bitcoin/bitcoin#23912: Insert and bump copyright headers
1362d6173f scripted-diff: Insert missed copyright headers (Hennadii Stepanov)
f47dda2c58 scripted-diff: Bump copyright headers (Hennadii Stepanov)
c29105efdc script: Fix copyright_header.py (Hennadii Stepanov)

Pull request description:

  This PR is an alternative to #23903.

  It bumps the existing copyright headers as we did every year, and adds the missed copyright headers.

  A small fix has been applied to the `copyright_header.py` in order to prevent such weird bumping as `2021` --> `2021-2017`.

ACKs for top commit:
  MarcoFalke:
    ACK 1362d6173f

Tree-SHA512: 204d970fe8c51546b26b8f03fe4297db8a9bef5101df851540b7b9eddbd3a09677ee81fdd882c60937d732407f42c9883165bd978272200cff8f90190f075905
2021-12-31 12:08:49 +01:00
Hennadii Stepanov f47dda2c58
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-

Commits of previous years:
* 2020: fa0074e2d8
* 2019: aaaaad6ac9
2021-12-30 19:36:57 +02:00
Hennadii Stepanov ec7b7d4a36
ci: Enable the gui in the TSan build 2021-12-30 10:53:19 +02:00
fanquake 2da97b271b
ci: use GCC 8 when building packages in native_qt5 CI
Our minimum required GCC is GCC 8, and this change in required for
changes like #23839 which take advantage of flags introduced in that
version of GCC.

This should have been included as part of
182de7ba10.
2021-12-23 10:53:02 +08:00
MarcoFalke fa6a548f54
ci: Disable s390x gui tests for now 2021-12-17 17:09:51 +01:00
Sjors Provoost d8b705f1ca
test: previous releases: add v22.0 2021-12-16 12:41:45 +07:00
Sjors Provoost 8a57a06a50
test: previous releases: add v0.21.0 2021-12-16 12:41:44 +07:00
fanquake 1b76b18f8a
Merge bitcoin/bitcoin#23585: scripted-diff: Drop Darwin version for better maintainability
2f356a0ca8 scripted-diff: Drop Darwin version for better maintainability (Hennadii Stepanov)

Pull request description:

  After this PR, any macOS tools version bumping in the future will touch fewer files in the repo.

  Pointing a Darwin version for the `--host` system does not matter for the following reasons:

  - in terms of the resulted binaries, we should only care about the minimum supported macOS version which is a separated parameter in our build system.

  - in terms of the build system itself, the usage of the `$(host)` variable is self-consistent enough. Btw `$(host_os)` value already has the version dropped:
  ```
  $ make -C depends --no-print-directory print-host_os HOST=x86_64-apple-darwin19
  host_os=darwin
  ```

ACKs for top commit:
  gruve-p:
    ACK 2f356a0ca8
  promag:
    ACK 2f356a0ca8.
  fanquake:
    ACK 2f356a0ca8

Tree-SHA512: 374896ab0ba02b0d8b4b21431fe963bd213b0d09586e0898c13a4c5fa294c1b693f1b2c92880c245c4157c14217b4825b36522f461930477f4d2a727086ebb2a
2021-12-09 16:13:33 +08:00
Hennadii Stepanov 78a6bc6919
build, qt: Use Android NDK r23 LTS 2021-12-05 03:00:02 +02:00