Skip to content
GitLab
  1. 15 Sep, 2022 2 commits
  3. 07 Sep, 2022 3 commits
  5. 06 Sep, 2022 2 commits
  7. 02 Sep, 2022 2 commits
  9. 17 Aug, 2022 2 commits
    • Ed Morley's avatar
      Release v216 (#1358) · 20e198b9
      Ed Morley authored
      20e198b9
    • Ed Morley's avatar
      Update path rewriting to support setuptools v64's PEP660 editable install mode (#1357) · b2dfe739
      Ed Morley authored 
      On Heroku, the application source directory exists at a different path at build time (`/tmp/build_<hash>`), than it does at runtime (`/app`). As such, the buildpack has to perform path rewriting via `.profile.d` scripts at runtime, to ensure any packaging related absolute paths in the build output are rewritten to reference the new path. (Thankfully this awful path rewriting will no longer be necessary in the future with CNBs.)

Previously the only files this path rewriting needed to update were the `*.pth` and `*.egg-link` files in `site-packages` created by setuptools when performing editable installs.

However setuptools v64 added support for PEP660 based editable install hooks:
https://setuptools.pypa.io/en/latest/history.html#v64-0-0
https://peps.python.org/pep-0660/

This feature is only used for projects that have a `pyproject.toml`, and for such projects, [if the config is deemed complex enough](https://github.com/pypa/setuptools/blob/d03da04e024ad4289342077eef6de40013630a44/setuptools/command/editable_wheel.py#L359-L368), setuptools creates a new [finder script](https://github.com/pypa/setuptools/blob/23d455c532fca91e6f00aa5950000739b058b6e5/setuptools/command/editable_wheel.py#L740-L809) in `site-packages` that dynamically handles package resolution. (Simpler configs get a static `.pth` file, which works fine with our existing path rewriting.)

This new file embeds the absolute path of the source directory at build time, so must be rewritten too. It has a filename of form: `__editable___my_package_0_0_1_finder.py`

As such, this PR adds support for rewriting these files, along with updated test fixtures to provide coverage of `pyproject.toml` based editable installs (alongside the existing `setup.py` based test fixture).

Whilst writing the new test, I encountered a difference in behaviour with setuptool's new editable install mode, which meant the fixtures had to be nested inside a `packages/` directory in order to avoid an `ImportError` due to the fact that the Python buildpack currently sets `PYTHONPATH=/app` at runtime. See:
https://github.com/pypa/setuptools/issues/3535

Note:
- The Python buildpack doesn't yet globally install this newer setuptools v64 release, since by design it pins to a specific version to prevent upstream changes from breaking apps overnight. (The version was recently updated to 63.4.3 in #1344.)
- However, for packages that have a `pyproject.toml` pip uses the approach described in PEP518, which uses an [isolated build environment](https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolation

). This environment typically pulls in latest setuptools (though this is controllable by the package owner via `[build-system]` in `pyproject.toml`), overriding our pinned global setuptools install.
- As such, even though we're not using v64 globally, users can still be broken by the upstream release.

A big thanks to @mrcljx for the initial PR in #1355 on which this was based :-)

Closes #1355.
GUS-W-11608693.

Co-authored-by: default avatarMarcel Jackwerth <marceljackwerth@gmail.com>
      b2dfe739
  11. 16 Aug, 2022 4 commits
  13. 15 Aug, 2022 8 commits
  15. 02 Aug, 2022 3 commits
  17. 04 Jul, 2022 1 commit
  19. 14 Jun, 2022 1 commit
  21. 13 Jun, 2022 2 commits
    • Ed Morley's avatar
      Enable retries and connection timeouts when using `curl` (#1335) · c6ebbec5
      Ed Morley authored 
      In the shimmed CNBs used in `heroku/builder` we have been seeing
quite a few transient errors related to buildpacks downloading from S3.

Adding appropriate retries and connection timeouts to all of our
buildpack's curl usages should help with these, as well as make builds
more reliable in general for users on Heroku, plus also anyone using a
shimmed CNB locally with Pack CLI (where the network connection may
be even less reliable).

The `--retry-connrefused` has been used since otherwise curl doesn't
retry cases where the connection was refused. Ideally we would use
`--retry-all-errors` which takes that one step further, however that
option was only added in curl 7.71, so is only supported by Heroku-22+.

I've intentionally not added `--max-time` to usages within buildpack
compile, since otherwise users running the buildpack locally on slow
connections may permanently hit timeouts (and timeouts mid-download
occur so rarely it's not worth the hassle of making this configurable).

For more on curl options, see:
https://curl.se/docs/manpage.html

GUS-W-11283397.
      c6ebbec5
    • Ed Morley's avatar
      Switch to the recommended regional S3 domain (#1334) · 2b0805e0
      Ed Morley authored 
      Whilst the global S3 endpoint (`s3.amazonaws.com`) still works,
AWS now recommends using the appropriate regional endpoint
to access the bucket:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#s3-legacy-endpoints

Our buildpack buckets are in `us-east-1`, whose regional domain
is `*.s3.us-east-1.amazonaws.com`:
https://docs.aws.amazon.com/general/latest/gr/s3.html#s3_region

GUS-W-11283397.
      2b0805e0
  23. 07 Jun, 2022 2 commits
  25. 06 Jun, 2022 1 commit
  27. 02 Jun, 2022 2 commits
  29. 01 Jun, 2022 2 commits
  31. 24 May, 2022 1 commit
    • Ed Morley's avatar
      Add support for the Heroku-22 stack (#1299) · 894e42c7
      Ed Morley authored 
      Adds binary generation and testing support for Heroku-22.

Runtime version support for Heroku-22 is as follows:
* Python 3.6 and older: Not supported since EOL.
* Python 3.7/3.8: Not supported since:
   - they are in the security-fix-only phase of their lifecycle
   - they reach EOL in 2023-06 and 2024-10 respectively, which is well
     before the EOL date of Heroku-22
   - upgrading from 3.7/3.9 to 3.9 is in most cases trivial, and
     so we should be encouraging users to upgrade.
* Python 3.9: Only 3.9.12+ is supported.
* Python 3.10: Only 3.10.4+ is supported.
* PyPy: Not supported, since is deprecated, was only ever a beta
   feature, the PyPy versions available for Heroku-18/20 are
   significantly out of date, and usage is virtually non-existent.

The `force_absolute_paths_buildpack` usage in tests has been removed
since:
* Heroku-22 no longer includes Ruby in the stack image, and that
   buildpack was implemented in Ruby rather than bash.
* That buildpack doesn't really add any value over just checking that
   things we put on the `PATH` (such as Python) can actually be used,
   which the existing tests already do.

GUS-W-10343718.
      894e42c7
  33. 18 May, 2022 2 commits