this post was submitted on 06 Nov 2024
193 points (98.0% liked)

Programming

17352 readers
309 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]



founded 1 year ago
MODERATORS
 

So I'm no expert, but I have been a hobbyist C and Rust dev for a while now, and I've installed tons of programs from GitHub and whatnot that required manual compilation or other hoops to jump through, but I am constantly befuddled installing python apps. They seem to always need a very specific (often outdated) version of python, require a bunch of venv nonsense, googling gives tons of outdated info that no longer works, and generally seem incredibly not portable. As someone who doesn't work in python, it seems more obtuse than any other language's ecosystem. Why is it like this?

(page 2) 50 comments
sorted by: hot top controversial new old
[–] [email protected] 23 points 2 days ago

Python never had much of a central design team. People mostly just scratched their own itch, so you get lots of different tools that do only a small part each, and aren't necessarily compatible.

[–] [email protected] 17 points 1 day ago

I agree. Python is my language of choice 80% or so of the time.

But my god, it does packaging badly! Especially if it's dependent on linking to compiled code!

Why it is like that, I couldn't tell. The language is older than git, so that might be part of it.

However, you're installing python libraries from github? I very very rarely have to do that. In what context do you have to do that regularly?

[–] atzanteol 12 points 1 day ago (1 children)

With all the hype surrounding Python it's easy to forget that it's a really old language. And, in my opinion, the leadership is a bit of a mess so there hasn't been any concerted effort on standardizing tooling.

Some unsolicited advice from somebody who is used more refined build environments but is doing a lot of Python these days:

The whole venv thing isn't too bad once you get the hang of it. But be prepared for people to tell you that you're using the wrong venv for reasons you'll never quit understand or likely need to care about. Just use the bundled "python -m venv venv" and you'll be fine despite other "better" alternatives. It's bundled so it's always available to you. And feel free to just drop/recreate your venv whenever you like or need. They're ephemeral and pretty large once you've installed a lot of things.

Use "pipx" to install python applications you want to use as programs rather than libraries. It creates and manages venvs for them so you don't get library conflicts. Something like "pip-tools" for example (pipx install pip-tools).

Use "pyenv" to manage installed python versions - it's a bit like "sdkman" for the JVM ecosystem and makes it easy to deal with the "specific versions of python" stuff.

For dependencies for an app - I just create a requirements.txt and "pip install -r requirements.txt" for the most part... Though I should use one of the 80 better ways to do it because they can help with updating versions automatically. Those tools mostly also just spit out a requirements.txt in the end so it's pretty easy to migrate to them. pip-tools is what my team is moving towards and it seems a reasonable option. YMMV.

load more comments (1 replies)
[–] [email protected] 8 points 1 day ago* (last edited 1 day ago) (1 children)

venv nonsense

I mean, the fact that it isn't more end-user invisible to me is annoying, and I wish that it could also include a version of Python, but I think that venv is pretty reasonable. It handles non-systemwide library versioning in what I'd call a reasonably straightforward way. Once you know how to do it, works the same way for each Python program.

Honestly, if there were just a frontend on venv that set up any missing environment and activated the venv, I'd be fine with it.

And I don't do much Python development, so this isn't from a "Python awesome" standpoint.

[–] [email protected] 2 points 1 day ago

pyenv and uv let you install and switch between multiple Python versions.

As for uv, those come from the Python build standalone project, if I remember correctly, pyenv also installs from there, but don't quote me on that.

[–] [email protected] 2 points 1 day ago

Yep, they are not portable, every app should come bundled with its own interpreter. As to why, I think historically it didn't target production grade application development.

[–] priapus 5 points 1 day ago (2 children)

Yeah the tooling sucks. The only tooling I've liked is Poetry, I never have trouble installing or packaging the apps that use it.

[–] [email protected] 1 points 1 day ago

Personally, I've found Poetry somewhat painful for developing medium-sized or larger applications (which I guess Python really isn't made for to begin with, but yeah).

Big problem is that its dependency resolution is probably a magnitude slower than it should be. Anytime we changed something about the dependencies, you'd wait for more than a minute on its verdict. Which is particularly painful, when you have to resolve version conflicts.

Other big pain point is that it doesn't support workspaces or multi-project builds or whatever you want to call them, so where you can have multiple related applications or libraries in the same repo and directly depending on each other, without needing to publish a version of the libraries each time you make a change.

When we started our last big Python project, none of the Python tooling supported workspaces out of the box. Now, there's Rye, which does so. But yeah, I don't have experience yet, with how well it works.

load more comments (1 replies)
[–] [email protected] 5 points 1 day ago* (last edited 1 day ago)

This is exactly how I feel about python as well... IMHO, it's good for some advanced stuff, where bash starts to hit its limits, but I'd never touch it otherwise

[–] [email protected] 2 points 1 day ago (1 children)

Docker might be solution here.

But from my experience most python scripts are absolute junk. The barrier for entry is low so there's a massive disparity in quality.

load more comments (1 replies)
[–] [email protected] 1 points 1 day ago (1 children)

I'm not sure this can be really fixed with Python 3, maybe we just have to hope for Python 4

[–] [email protected] -1 points 1 day ago (4 children)

It's fixed, and the python version had nothing to do with it. Just use hatch

load more comments (4 replies)
[–] [email protected] 3 points 1 day ago

It... depends. There is some great tooling for Python -- this was less true only a few years ago, mind you -- but the landscape is very much in flux, and usage of the modern stuff is not yet widespread. And a lot of the legacy stuff has a whole host of pitfalls.

Things are broadly progressing in the right direction, and I'd say I'm cautiously optimistic, although if you have to deal with anything related to conda then for the time being: good luck, and sorry.

[–] [email protected] 0 points 1 day ago* (last edited 1 day ago) (1 children)

Just out of curiosity, I haven't seen anyone recommend miniconda... Why so, is there something wrong I'm not aware of?

I'm no expert, but I totally feel you, python packages, dependencies and version matching is a real nightmare. Even with venv I had a hard time to make everything work flawlessly, especially on MacOS.

However, with miniconda everything was way easier to configure and worked as expected.

load more comments (1 replies)
[–] [email protected] 2 points 1 day ago (2 children)

Tried to install Automatic1111 for Stable Diffusion in an Arch distrobox, and despite editing the .sh file to point to the older tarballed Python version as advised on Github, it still tells me it uses the most up to date one that's installed system wide and thus can't install pytorch. And that's pretty much where my personal knowledge ends, and apparently that of those (i.e. that one person) on Github. ¯\_(ツ)_/¯

Always funny when people urge you to ask for help but no one ends up actually helping.

[–] zaphodb2002 1 points 1 day ago (1 children)

Lol this is exactly why I made this post. I ended up using ComfyUI instead which has other, different python issues, but I got it working (kinda, no GPU but it's fine it works)

[–] [email protected] 1 points 1 day ago

I definitely want gpu support. Although I struggle with that somewhat on Koboldcpp as well where I can't use ROCm, only Vulkan. Unsure where the difference is performance wise.

I'd like to try the other UIs too, but the problem is that Automatic1111 is where the majority of additional plugins can be found.

[–] [email protected] 1 points 1 day ago* (last edited 1 day ago)

despite editing the .sh file to point to the older tarballed Python version as advised on Github, it still tells me it uses the most up to date one that's installed system wide and thus can't install pytorch.

Can you paste your commands and output?

If you want, maybe on [email protected], since I think that people seeing how to get Automatic1111 set up might help others.

I've set it up myself, and I don't mind taking a stab at getting it working, especially if it might help get others over the hump to a local Automatic1111 installation.

[–] [email protected] 1 points 1 day ago (1 children)

I'm no Python expert either and yeah, from an outsider's perspective it seems needlessly confusing. easy_install that's never been easy, pip that should absolutely be put on a Performance Improvement Plan, and now this venv nonsense.

You can criticize javascript's ridiculous dependencies all you want (left-pad?), but one thing that they absolutely got right is how to manage them. Everything's in node_modules and that's it. Yeah, you might get eleven copies of left-pad on your system, but you know what you NEVER get? Version conflicts between projects you're working on.

[–] [email protected] 1 points 1 day ago

Seriously. Those are EXACTLY the thoughts I had after I was forced to deal with Python after a ton of time writing projects in JS.

[–] [email protected] -1 points 1 day ago* (last edited 1 day ago)

Difficult? How so? I find compiling C and C++ stuff much more difficult than anything python. It never works on the first try whereas with python the chances are much much higher.

What's is so difficult to understand about virtual envs? You have global python packages, you can also have per user python packages, and you can create virtual environments to install packages into. Why do people struggle to understand this?

The global packages are found thanks to default locations, which can be overridden with environment variables. Virtual environments set those environment variables to be able to point to different locations.

python -m venv .venv/ means python will execute the module venv and tell it to create a virtual environment in the .venv folder in the current directory. As mentioned above, the environment variables have to be set to actually use it. That's when source .venv/bin/activate comes into play (there are other scripts for zsh and fish). Now you can run pip install $package and then run the package's command if it has one.

It's that simple. If you want to, you can make it difficult by doing sudo pip install $package and fucking up your global packages by possibly updating a dependency of another package - just like the equivalent of updating glibc from 1.2 to 1.3 and breaking every application depending on 1.2 because glibc doesn't fucking follow goddamn semver.

As for old versions of python, bro give me a break. There's pyenv for that if whatever old ass package you're installing depends on an ancient 10 year old python version. You really think building a C++ package from 10 years ago will work more smoothly than python? Have fun tracking down all the unlocked dependency versions that "Worked On My Machine 🏧" at the start of the century.

The only python packages I have installing are those with C/C++ dependencies which have to be compiled at install time.

Y'all have got to be meme'ing.

Anti Commercial-AI license

load more comments
view more: ‹ prev next ›