this post was submitted on 06 Nov 2024
202 points (98.1% liked)

Programming

17492 readers
17 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]



founded 1 year ago
MODERATORS
 

So I'm no expert, but I have been a hobbyist C and Rust dev for a while now, and I've installed tons of programs from GitHub and whatnot that required manual compilation or other hoops to jump through, but I am constantly befuddled installing python apps. They seem to always need a very specific (often outdated) version of python, require a bunch of venv nonsense, googling gives tons of outdated info that no longer works, and generally seem incredibly not portable. As someone who doesn't work in python, it seems more obtuse than any other language's ecosystem. Why is it like this?

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 70 points 2 weeks ago

Python's packaging is not great. Pip and venvs help but, it's lightyears behind anything you're used to. My go-to is using a venv for everything.

[–] [email protected] 59 points 2 weeks ago (8 children)

It's something of a "14 competing standards" situation, but uv seems to be the nerd favourite these days.

[–] [email protected] 30 points 2 weeks ago (3 children)

I still do the python3 -m venv venv && source venv/bin/activate

How can uv help me be a better person?

[–] [email protected] 13 points 2 weeks ago (1 children)

And pip install -r requirements.txt

[–] [email protected] 13 points 2 weeks ago (3 children)

Fuck it, I just use sudo and live with the consequences.

[–] [email protected] 7 points 2 weeks ago
load more comments (2 replies)
[–] [email protected] 7 points 2 weeks ago (1 children)
  1. let pyproject.toml track the dependencies and dev-dependencies you actually care about
  • dependencies are what you need to run your application
  • dev-dependencies are not necessary to run your app, but to develop it (formatting, linting, utilities, etc)
  1. it can track exactly what's needed ot run the application via the uv.lock file that contains each and every lib that's needed.
  2. uv will install the needed Python version for you, completely separate from what your system is running.
  3. uv sync and uv run <application> is pretty much all you need to get going
  4. it's blazingly fast in everything
load more comments (1 replies)
[–] [email protected] 3 points 2 weeks ago

If you’re happy with your solution, that’s great!

uv combines a bunch of tools into one simple, incredibly fast interface, and keeps a lock file up to date with what’s installed in the project right now. Makes docker and collaboration easier. Its main benefit for me is that it minimizes context switching/cognitive load

Ultimately, I encourage you to use what makes sense to you tho :)

load more comments (7 replies)
[–] [email protected] 53 points 2 weeks ago

Python is the only programming language that has forced me to question what the difference is between an egg and a wheel.

[–] [email protected] 50 points 2 weeks ago (1 children)

You re not stupid, python's packaging & versionning is PITA. as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem

[–] [email protected] 16 points 2 weeks ago

as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem

A perfect summary of the history of computer code!

[–] [email protected] 36 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

No, it's not just you, Python's tooling is a mess. It's not necessarily anyone's fault, but there are a ton of options and a lot of very similarly named things that accomplish different (but sometimes similar) tasks. (pyenv, venv, and virtualenv come to mind.) As someone who considers themselves between beginner and intermediate proficiency in Python, this is my biggest hurdle right now.

[–] [email protected] 15 points 2 weeks ago (1 children)

Python’s tooling is a mess.

Not only that. It's a historic mess. Over the years, growing a better and better toolset left a lot of projects in a very messy state. So many answers on Stack Overflow that mention easy_install - I still don't know what it is, but I guess it was some kind of proto uv.

[–] [email protected] 6 points 2 weeks ago

Every time I'm doing anything with Python I ask myself if Java's tooling is this complicated or I'm just used to it by now. I think a big part of the weirdness is that a lot of Python tooling is tied to the Python installation whereas in Java things like Maven and Gradle are separate. In addition, I think dependencies you install get tied to that Python installation, while in Java they just are in a cache for Maven/Gradle. And in the horrible scenario where you need to use different versions of Maven/Gradle (one place I was at specifically needed Maven 3.0.3 for one project and a different for a different, don't ask, it's dumb and their own fault for setting it up that way) at least they still have one common cache for everything.

I guess it also helps that with Java you (often) don't need platform specific jar files. But Python is often used as an easy and dynamic scripting interface over more performant, native code. So you don't really run into things like "this artifact doesn't have a 64 bit arm version for python 2" often with Java. But that's not a fault of Python's tooling, it's just the reality of how it's used.

[–] [email protected] 30 points 2 weeks ago (4 children)

Yes it's terrible. The only hope on the horizon is uv. It's significantly better than all the other tooling (Poetry, pip, pipenv, etc.) so I think it has a good chance of reducing the options to just Pip or uv at least.

But I fully expect the Python Devs to ignore it, and maybe even make life deliberately difficult for it like they did for static analysers. They have some strange priorities sometimes.

[–] [email protected] 9 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

I like the idea of uv, but I hate the name. Libuv is already a very popular C library, and used in everything from NodeJS to Julia to Python (through the popular uvloop module). Every time I see someone mention uv I get confused and think they're talking about uvloop until I remember the Astral project, and then reconfirm to myself how much I disapprove of their name choice.

load more comments (1 replies)
load more comments (3 replies)
[–] [email protected] 26 points 2 weeks ago* (last edited 2 weeks ago) (7 children)

Python developer here. Venv is good, venv is life. Every single project I create starts with

python3 -m venv venv

source venv/bin/activate

pip3 install {everything I need}

pip3 freeze > requirements.txt

Now write code!

Don't forget to update your requirements.txt using pip3 freeze again anytime you add a new library with pip.

If you installed a lot of packages before starting to develop with virtual environments, some libraries will be in your OS python install and won't be reflected in pip freeze and won't get into your venv. This is the root of all evil. First of all, don't do that. Second, you can force libraries to install into your venv despite them also being in your system by installing like so:

pip3 install --ignore-installed mypackage

If you don't change between Linux and windows most libraries will just work between systems, but if you have problems on another system, just recreate the whole venv structure

rm -rf venv (...make a new venv, activate it) pip3 install -r requirements.txt

Once you get the hang of this you can make Python behave without a lot of hassle.

This is a case where a strength can also be a weakness.

[–] [email protected] 20 points 2 weeks ago (1 children)

pip3 freeze > requirements.txt

I hate this. Because now I have a list of your dependencies, but also the dependencies of the dependencies, and I now have regular dependencies and dev-dependencies mixed up. If I'm new to Python I would have NO idea which libraries would be the important ones because it's a jumbled mess.

I've come to love uv (coming from poetry, coming from pip with a requirements/base.txt and requirements/dev.txt - gotta keep regular dependencies and dev-dependencies separate).

uv sync

uv run <application>

That's it. I don't even need to install a compatible Python version, as uv takes care of that for me. It'll automatically create a local .venv/, and it's blazingly fast.

load more comments (1 replies)
[–] [email protected] 7 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

Okay, now give me those steps but what to do if I clone an already existing repo please

load more comments (1 replies)
[–] [email protected] 6 points 2 weeks ago

OP sounds like a victim of Python 3, finding various Python 2 projects on the internet, a venv isn't going to help

[–] [email protected] 6 points 2 weeks ago

You have been in lala land for too long. That list of things to do is insane. Venv is possibly one of the worst solutions around, but many Python devs are incapable of seeing how bad it is. Just for comparison, so you can understand, in Ruby literally everything you did is covered by one command bundle. On every system.

load more comments (3 replies)
[–] [email protected] 23 points 2 weeks ago (2 children)

everyone focuses on the tooling, not many are focusing on the reason: python is extremely dynamic. like, magic dynamic you can modify a module halfway through an import, you can replace class attributes and automatically propagate to instances, you can decompile the bytecode while it's running.

combine this with the fact that it's installed by default and used basically everywhere and you get an environment that needs to be carefully managed for the sake of the system.

js has this packaging system down pat, but it has the advantage that it got mainstream in a sandboxed isolated environment before it started leaking out into the system. python was in there from the beginning, and every change breaks someone's workflow.

the closest language to look at for packaging is probably lua, which has similar issues. however since lua is usually not a standalone application platform it's not a big deal there.

load more comments (2 replies)
[–] [email protected] 23 points 2 weeks ago

Python never had much of a central design team. People mostly just scratched their own itch, so you get lots of different tools that do only a small part each, and aren't necessarily compatible.

[–] WolfLink 20 points 2 weeks ago* (last edited 2 weeks ago)

The reason you do stuff in a venv is to isolate that environment from other python projects on your system, so one Python project doesn’t break another. I use Docker for similar reasons for a lot of non-Python projects.

A lot of Python projects involve specific versions of libraries, because things break. I’ve had similar issues with non-Python projects. I’m not sure I’d say Python is particularly worse about it.

There are tools in place that can make the sharing of Python projects incredibly easy and portable and consistent, but I only ever see the best maintained projects using them unfortunately.

[–] [email protected] 17 points 2 weeks ago

I agree. Python is my language of choice 80% or so of the time.

But my god, it does packaging badly! Especially if it's dependent on linking to compiled code!

Why it is like that, I couldn't tell. The language is older than git, so that might be part of it.

However, you're installing python libraries from github? I very very rarely have to do that. In what context do you have to do that regularly?

[–] [email protected] 17 points 2 weeks ago (1 children)

Python is hacky, because it hacks. There’s a bunch of ways you can do anything. You can run it on numerous platforms, or even on web assembly. It’s not maintained centrally. Each “app” you find is just somebodies hack project they’re sharing with you for fun.

[–] [email protected] 11 points 2 weeks ago (3 children)
[–] [email protected] 4 points 2 weeks ago (2 children)

After using python, I'm of the opinion that perl was much cleaner.

[–] [email protected] 3 points 2 weeks ago

Nothing comes close to Perl’s abuse of global variables. Oh you called this function? Take a guess which global variables it will use.

load more comments (1 replies)
load more comments (2 replies)
[–] [email protected] 14 points 2 weeks ago (7 children)
[–] Jocker 13 points 2 weeks ago

I've started using poetry and the experience has improved.

[–] [email protected] 13 points 2 weeks ago (1 children)

The venv stuff is pretty annoying, I agree.

[–] [email protected] 3 points 2 weeks ago (1 children)

As a baby Python Dev, I'm glad it's not just me.

[–] [email protected] 5 points 2 weeks ago (1 children)

I've been full time writing python professionally since 2015. You get used to it. It starts to just make sense and feel normal. Then when you move to a different language environment you wonder why their tooling doesn't use a virtualenv.

load more comments (1 replies)
[–] atzanteol 12 points 2 weeks ago (1 children)

With all the hype surrounding Python it's easy to forget that it's a really old language. And, in my opinion, the leadership is a bit of a mess so there hasn't been any concerted effort on standardizing tooling.

Some unsolicited advice from somebody who is used more refined build environments but is doing a lot of Python these days:

The whole venv thing isn't too bad once you get the hang of it. But be prepared for people to tell you that you're using the wrong venv for reasons you'll never quit understand or likely need to care about. Just use the bundled "python -m venv venv" and you'll be fine despite other "better" alternatives. It's bundled so it's always available to you. And feel free to just drop/recreate your venv whenever you like or need. They're ephemeral and pretty large once you've installed a lot of things.

Use "pipx" to install python applications you want to use as programs rather than libraries. It creates and manages venvs for them so you don't get library conflicts. Something like "pip-tools" for example (pipx install pip-tools).

Use "pyenv" to manage installed python versions - it's a bit like "sdkman" for the JVM ecosystem and makes it easy to deal with the "specific versions of python" stuff.

For dependencies for an app - I just create a requirements.txt and "pip install -r requirements.txt" for the most part... Though I should use one of the 80 better ways to do it because they can help with updating versions automatically. Those tools mostly also just spit out a requirements.txt in the end so it's pretty easy to migrate to them. pip-tools is what my team is moving towards and it seems a reasonable option. YMMV.

load more comments (1 replies)
[–] [email protected] 8 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

venv nonsense

I mean, the fact that it isn't more end-user invisible to me is annoying, and I wish that it could also include a version of Python, but I think that venv is pretty reasonable. It handles non-systemwide library versioning in what I'd call a reasonably straightforward way. Once you know how to do it, works the same way for each Python program.

Honestly, if there were just a frontend on venv that set up any missing environment and activated the venv, I'd be fine with it.

And I don't do much Python development, so this isn't from a "Python awesome" standpoint.

load more comments (1 replies)
[–] priapus 6 points 2 weeks ago (2 children)

Yeah the tooling sucks. The only tooling I've liked is Poetry, I never have trouble installing or packaging the apps that use it.

load more comments (2 replies)
[–] [email protected] 4 points 2 weeks ago* (last edited 2 weeks ago)

This is exactly how I feel about python as well... IMHO, it's good for some advanced stuff, where bash starts to hit its limits, but I'd never touch it otherwise

[–] [email protected] 3 points 2 weeks ago

It... depends. There is some great tooling for Python -- this was less true only a few years ago, mind you -- but the landscape is very much in flux, and usage of the modern stuff is not yet widespread. And a lot of the legacy stuff has a whole host of pitfalls.

Things are broadly progressing in the right direction, and I'd say I'm cautiously optimistic, although if you have to deal with anything related to conda then for the time being: good luck, and sorry.

load more comments
view more: next ›