How to improve Python packaging, or why fourteen tools are at least twelve too many
How to improve Python packaging, or why fourteen tools are at least twelve too many
A journey to the world of Python packaging, a visit to the competition, a hopeful look at the future, and highlights from a disappointing discussion.
I stumbled upon this while researching package management options for python, and found it a really interesting read.
I like python as a language but this mess is something that needs to be addressed for me to consider python for future projects. I can't imagine how confusing it must be for new users.
It's worth noting that there are basically just 3 systems worth considering, maybe even just 2.
pip is usually part of the python distribution, so any lightweight project can be finished in 1-5 minutes with pip. It's also quite widespread and the vast majority of publishers (if not all) target pip compatibility.
Poetry is a great project management framework and it deals with dependency management beautifully. If you're doing any data engineering or backend development, for any project that has more than 1 dependency and 200 lines of code, then Poetry is probably the best tool to use. Poetry makes the whole mess with helper tools like pip-tools seem outdated.
Conda is for the crazy world of data science libraries where developers don't bother with compatibility too much. Conda does it for them. And the users of those libraries can benefit from using conda.
I think the big competition is between poetry and pip. Maybe one day poetry will come as part of some Python distributions.
Worth noting the thing that conda does well is handle system-level dependencies, there are a lot more binaries available there (beyond just Python packages) that make it easier to install packages without having to compile C++ code or something.
Besides that I hate it haha, it makes everything else so complicated
I’ve also not had great experiences with poetry, it’s supposed to use the lock file to lock in dependencies but whenever I did anything the lock file was always being updated which kind of defeats the point. I’m sure it was user error but the fact that the lock file doesn’t stay static by default is already weird to me
Our organisation has gone all in on Poetry, no regrets so far. The UX and dlscoverability is just so much better than the other options.
I do look jealously at languages that have great official tools like
goandcargothough.The value of cargo and go tools doesn’t come from the all-in-one nature of them, it comes from the official nature of them.
If something doesn’t work with cargo, it is a bug. Period. There isn’t any “it works with pip” back and forth arguing over whose fault it actually is (package? Or poetry/pipenv/pip-tools/conda/etc? This happened with pytorch a while ago, and I’m not sure if poetry and pytorch get along even now)
There also isn’t any debate over project files or configuration stuff — Pyproject.toml vs setup.cfg vs random dot files in the project directory — if you are a currently developed project you support whatever cargo supports and you move to support the latest format rather than dragging your feet for years (pyproject.toml has been the “next thing” for python since 2016! And is only finally getting widespread support now… 7 years later).
Yes I'm following Huak, it looks promising. But as Spott says, just because a tool exists, its not the same as having the tool which is fully supported, standardised and everyone uses.
IMO Python could have this but as the posted article discusses there is no movement or will to make it happen.
I love cargo, but cargo.io could REALLY make good use of namespaces. It's insane when clear library names are taken by highschoolers at their first project and there is nothing to be done about it. I'd also like some kind of curating on the packages.
It is quite telling that the one tool that covers most of the features expected from a packaging tool is one maintained by a single author: PDM.
In my experience it has been a pleasure to use it as well, even though it is quite new.
I have been using PDM for a while now and love it, can't believe I was using pip before, especially for venv handling and packaging.
I wonder, having tools like PDM or Poetry that unify the whole process, why do people prefer to use fragmented tools (one for venv, another for packaging, another for publishing)? Is it just entrenched habits or not knowing the benefits of other tools?
I'm only an intermediate python dev but pip and venv do the trick. pyenv with the virtualenv plugin is nice on my Mac machine but I don't have any such equivalent on my Windows machine, I just need to remember to enable the venv.
I used Poetry for some project but ran into some problems. It was a long time ago and I didn't really know enough about what was going on to really explain why it wasn't working well. Part of the problem was that it was on a Raspberry Pi Zero W so it was extremely slow and it is possible everything would've been fine on a more traditional setup. My understanding is that a lot of legacy tools don't define what their dependencies are in a strict way and it is a costly operation for Poetry to execute the code in the setup files or something like that. It took hours but I also think it was trying to build Python from source or something like that.
As a Python hobbyist pip and venv get what I need done and have the massive advantage of coming bundled with Python itself, but I understand that there are problems that Python professionals probably run into that aren't solved as easily.
I'm only an intermediate python dev but pip and venv do the trick. pyenv with the virtualenv plugin is nice on my Mac machine but I don't have any such equivalent on my Windows machine, I just need to remember to enable the venv.
I used Poetry for some project but ran into some problems. It was a long time ago and I didn't really know enough about what was going on to really explain why it wasn't working well. Part of the problem was that it was on a Raspberry Pi Zero W so it was extremely slow and it is possible everything would've been fine on a more traditional setup. My understanding is that a lot of legacy tools don't define what their dependencies are in a strict way and it is a costly operation for Poetry to execute the code in the setup files or something like that. It took hours but I also think it was trying to build Python from source or something like that.
As a Python hobbyist pip and venv get what I need done and have the massive advantage of coming bundled with Python itself, but I understand that there are problems that Python professionals probably run into that aren't solved as easily.