r/Python • • 2d ago

Discussion 684 PyPI packages run code at every Python start, and only 13 are ready for PEP 829

Python runs any line that starts with import in a .pth file in site-packages, every time it starts. You don't have to import the package. This year, the malicious releases of litellm (March and October) and agent-framework (November) all used a .pth file, so installing them was enough to run the credential stealer.

I wanted to see how many ordinary packages use the same feature. I took the latest release of 354,763 PyPI projects, found every .pth file with an import line, downloaded them and read them. The results:

  • 684 packages install a .pth file that runs on every Python start.
  • 244 of them run their own code: setuptools, coverage, coloredlogs, pywin32 and others. The other 440 only run old setuptools namespace boilerplate (-nspkg.pth).
  • coverage and coloredlogs do nothing unless you set an environment variable. urllib3-future copies its own files over your urllib3 folder at startup.
  • 52 packages put the file where Python never reads it on Linux or macOS, including files from temporary build folders that were packed by mistake.
  • Python 3.15 deprecates these import lines (PEP 829), and 3.18 will ignore them. Only 13 of the 244 packages already ship the replacement .start file.

The post includes a command that lists the .pth files with import lines in your own environment, and a table of what the 15 most downloaded ones do: https://py-code.org/blog/python-pth-files-pypi/

190 Upvotes

30 comments sorted by

90

u/zurtex 2d ago

PEP 829 was only approved earlier this year, Python 3.15 only just came out and all it does is deprecate .pth.

It's common for libraries not to migrate until things are about to break or just after they do break. Welcome to software maintenance.

17

u/billsil 2d ago

Exactly. Oh shit, my CI is failing because package x just pushed a new version. I’ll get to it. Nobody updates immediately and if they do they probably won’t hit this edge case.

7

u/Nangangagatngpokpok 1d ago

yeahh that's basically how most migrations happen. The deadline isn't what gets people moving, but it's the random CI failure on a tuesday morning

1

u/marr75 1d ago

Yep, if I was constantly trying to out-think my CI, it wouldn't be doing much for me.

3

u/Wonderful-Habit-139 2d ago

Rather migrating until things break, python libraries tend to support Python versions that are not EOL. Meaning as of today they tend to support Python version 3.11 and up.

2

u/that_baddest_dude 1d ago

Yeah I've got the extremely fun task of porting a bunch of data analyst code by amateurs who wrote it in python 3.8 pandas 1.2.5, never maintained it, and have since left the company. I'm not having a good time

38

u/michallandry62 2d ago

Worth saying for anyone skimming: .start files don't really fix the supply chain part. A malicious wheel can still ship one and get code running on every interpreter start. What changes is it has to be a named module:function entry point instead of arbitrary code crammed onto an import line, so it's a lot easier to grep for and audit. The real guard is still at install time (pinning with hashes, waiting a few days before taking new releases, actually looking at new deps).

The urllib3-future thing is wild though. A package overwriting another package's files at startup is exactly the kind of behavior I'd never think to check when debugging some weird urllib3 issue.

16

u/Individual-Flow9158 2d ago

They do declare that's the default, and almost the whole point of urllib3-future. https://urllib3future.readthedocs.io/en/latest/cohabitation.html#why-there-are-two-installation-modes

But I wouldn't have made that design decision. I'm sure they meant well, but overwriting core libraries doesn't feel like being a good citizen of user land to me.

Yet another reason to use a venv though.

5

u/michallandry62 1d ago

Fair point, it is documented, I should have said that. Agree on the design call though: when a package quietly takes over another library's import name, debugging gets miserable the moment two dependencies expect different behaviour. A venv per project has saved me from more of these than I'd like to admit.

2

u/Individual-Flow9158 1d ago

Thanks. You're entirely correct about the .start files by the way.

10

u/Nangangagatngpokpok 1d ago

this is one of those Python behaviors that don't know people exists until it shows up in supply chain attack. Kind crazy that hundreds of packages can run code on startup and only 13 have already moved to the replacement mechanism, esp with PEP 829 giving plenty of warning

5

u/JanEric1 1d ago

I mean the old stuff will still work until (including) 3.17

So there is really no reason to hurry. People will also still need to have the old way in until python 3.19 is released because thats when python 3.14 goes eol and if you want to support all non-eol versions you need to keep it around until then (and use both)

1

u/Nangangagatngpokpok 1d ago

that's good, from a maintainer's POV, shipping both mechanisms for several yrs is probably the practical choice, so it's not surprising that adoption looks low this early in the deprecation cycle

7

u/shinitakunai 2d ago edited 2d ago

Wouldn't the same happen with .start file?

6

u/Smallpaul 2d ago

You sent me on a real wild goose chase trying to figure out what a .starr file is.

6

u/shinitakunai 2d ago

A typo 😅

-1

u/finalsight 2d ago

Sounds like you have to actually use the imported package for it to start running its code, rather than it executing just by existing when you start python

5

u/Individual-Flow9158 2d ago

That's not true:

Note: Executable lines (import lines in name.pth files and name.start file entry points) are always run at Python startup (unless -S is given to disable the site.py module entirely), regardless of whether a particular module is actually going to be used.

https://docs.python.org/3/library/site.html#site-start-files

7

u/RedEyed__ 2d ago

How about code execution during pip install?

7

u/Individual-Flow9158 2d ago

If you use pip install --only-binary then it should only unzip wheels, and not do any building from source that requires executing code?

2

u/nekokattt 2d ago

how does that work if you have a setup.py rather than a pyproject.toml?

2

u/Individual-Flow9158 1d ago

If the package needs setup.py to be run on every user's machine, then it cannot ship one static wheel for every user on each platform, and such a setup.py cannot be replaced by pyproject.toml alone (something like hatch's build hooks are needed).

2

u/Nangangagatngpokpok 1d ago

that's a good point but packages that need environment specific builds steps still need some mechanism to execute code, whether that's build hooks or a custom backend

3

u/Individual-Flow9158 1d ago

It's misleading for users if such packages ship wheels then. They shouldn't do so, in my personal opinion.

2

u/Nangangagatngpokpok 1d ago

idk if i go that far, wheels are still valuable for users who can use prebuilt package, even if some edge cases require a source build. The important part is being clear about when a wheel is sufficient and when extra build steps are expected

1

u/RedEyed__ 2d ago

Didn’t know that.
But it won’t work if the package requires executing code while installing.

2

u/Individual-Flow9158 1d ago

I'm not sure how such a package can build pre-installed wheels.

1

u/patryk_builds 22h ago

Good timing for me, i published my first package to pypi today and had no idea a wheel could do this. i checked what's inside mine before uploading code, py.typed, licence, but only because i was worried about shipping the wrong files. does anything in here run at interpreter start wasn't a question i knew to ask.The 52 packages that put the file where python never reads it are the funny part. dead on arrival, and probably nobody noticed, because the feature silently not working looks the same as it working.

Two questions:

- The 440 with only the old namespace boilerplate: are those all abandoned, or do they include packages people still install a lot? if they're still popular, 3.18 breaks namespace imports for them without anyone having done anything wrong recently

- 13 of 244 is about 5%. did you open issues on the big ones, or is the plan to let the deprecation warning in 3.15 do the work?

Going to run the command from your post on my own environment tonight

1

u/Business-Feedback635 19h ago

Thanks, and congrats on the first package.

The 440: not all abandoned. 172 of them are among the 60,000 most downloaded projects, about 88M downloads a month together. The biggest is google-cloud-aiplatform at about 71M a month. I haven't tested whether they break without the lines on 3.18, since implicit namespace packages may cover them.

-10

u/bedel99 2d ago

Wow some one discovered AI