this post was submitted on 15 Jan 2025

80 points (97.6% liked)

Programming

17984 readers

113 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]

founded 2 years ago

MODERATORS

[email protected]

Am I crazy in thinking that bash is good enough for production? (lemm.ee)

submitted 2 weeks ago by [email protected] to c/[email protected]

120 comments fedilink hide all child comments

This may make some people pull their hair out, but I’d love to hear some arguments. I’ve had the impression that people really don’t like bash, not from here, but just from people I’ve worked with.

There was a task at work where we wanted something that’ll run on a regular basis, and doesn’t do anything complex aside from reading from the database and sending the output to some web API. Pretty common these days.

I can’t think of a simpler scripting language to use than bash. Here are my reasons:

Reading from the environment is easy, and so is falling back to some value; just do ${VAR:-fallback}; no need to write another if-statement to check for nullity. Wanna check if a variable’s set to something expected? if [[ <test goes here> ]]; then <handle>; fi
Reading from arguments is also straightforward; instead of a import os; os.args[1] in Python, you just do $1.
Sending a file via HTTP as part of an application/x-www-form-urlencoded request is super easy with curl. In most programming languages, you’d have to manually open the file, read them into bytes, before putting it into your request for the http library that you need to import. curl already does all that.
Need to read from a curl response and it’s JSON? Reach for jq.
Instead of having to set up a connection object/instance to your database, give sqlite, psql, duckdb or whichever cli db client a connection string with your query and be on your way.
Shipping is… fairly easy? Especially if docker is common in your infrastructure. Pull Ubuntu or debian or alpine, install your dependencies through the package manager, and you’re good to go. If you stay within Linux and don’t have to deal with differences in bash and core utilities between different OSes (looking at you macOS), and assuming you tried to not to do anything too crazy and bring in necessary dependencies in the form of calling them, it should be fairly portable.

Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.

For most bash gotchas, shellcheck does a great job at warning you about them, and telling how to address those gotchas.

There are probably a bunch of other considerations but I can’t think of them off the top of my head, but I’ve addressed a bunch before.

So what’s the dealeo? What am I missing that may not actually be addressable?

top 50 comments

sorted by: hot top controversial new old

[–] [email protected] 39 points 2 weeks ago (1 children)

I just don't think bash is good for maintaining the code, debugging, growing the code over time, adding automated tests, or exception handling

[–] [email protected] 9 points 2 weeks ago (2 children)

If you need anything that complex and that it’s critical for, say, customers, or people doing things directly for customers, you probably shouldn’t use bash. Anything that needs to grow? Definitely not bash. I’m not saying bash is what you should use if you want it to grow into, say, a web server, but that it’s good enough for small tasks that you don’t expect to grow in complexity.

[–] [email protected] 23 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

it’s (bash) good enough for small tasks that you don’t expect to grow in complexity.

I don't think you'll get a lot of disagreement on that, here. As mention elsewhere, my team prefers bash for simple use cases (and as their bash-hating boss, I support and agree with how and when they use bash.)

But a bunch of us draw the line at database access.

Any database is going to throw a lot of weird shit at the bash script.

So, to me, a bash script has grown to unacceptable complexity on the first day that it accesses a database.

[–] [email protected] 4 points 2 weeks ago (1 children)

We have dozens of bash scripts running table cleanups and maintenece tasks on the db. In the last 20 years these scripts where more stable than the database itself (oracle -> mysql -> postgres).

But in all fairness they just call the cliclient with the appropiate sql and check for the response code, generating a trap.

[–] [email protected] 3 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

That's a great point.

I post long enough responses already, so I didn't want to get into resilience planning, but your example is a great highlight that there's rarely hard and fast rules about what will work.

There certainly are use cases for bash calling database code that make sense.

I don't actually worry much when it's something where the first response to any issue is to run it again in 15 minutes.

It's cases where we might need to do forensic analysis that bash plus SQL has caused me headaches.

[–] [email protected] 3 points 2 weeks ago (1 children)

Yeah, if it feels like a transaction would be helpful, at least go for pl/sql and save yourself some pain. Bash is for system maintenance, not for business logic.

Heck, I wrote a whole monitoring system for a telephony switch with nothing more than bash and awk and it worked better than the shit from the manufacturer, including writing to the isdn cards for mobile messaging. But I wouldn't do that again if I have an alternative.

[–] [email protected] 2 points 2 weeks ago

Bash is for system maintenance, not for business logic.

That is such a good guiding principle. I'm gonna borrow that.

[–] [email protected] 9 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

small tasks that you don’t expect to grow in complexity

On one conference I heard saying: "There is no such thing as temporary solution and there is no such thing as proof of concept". It's an overexaguration of course but it has some truth to it - there's a high chance that your "small change" or PoC will be used for the next 20 years so write it as robust and resilient as possible and document it. In other words everything will be extended, everything will be maintained, everything will change hands.

So to your point - is bash production ready? Well, depends. Do you have it in git? Is it part of some automation pipeline? Is it properly documented? Do you by chance have some tests for it? Then yes, it's production ready.

If you just "write this quick script and run it in cron" then no. Because in 10 years people will pull their hair screaming "what the hell is hapenning?!"

Edit: or worse, they'll scream it during the next incident that'll happen at 2 AM on Sunday

load more comments (1 replies)

[–] [email protected] 37 points 2 weeks ago

"Use the best tool for the job, that the person doing the job is best at." That's my approach.

I will use bash or python dart or whatever the project uses.

[–] [email protected] 27 points 2 weeks ago

Honestly, if a script grows to more than a few tens of lines I'm off to a different scripting language because I've written enough shell script to know that it's hard to get right.

Shellcheck is great, but what's greater is a language that doesn't have as many gotchas from the get go.

[–] [email protected] 24 points 2 weeks ago (8 children)

I'm afraid your colleagues are completely right and you are wrong, but it sounds like you genuinely are curious so I'll try to answer.

I think the fundamental thing you're forgetting is robustness. Yes Bash is convenient for making something that works once, in the same way that duct tape is convenient for fixes that work for a bit. But for production use you want something reliable and robust that is going to work all the time.

I suspect you just haven't used Bash enough to hit some of the many many footguns. Or maybe when you did hit them you thought "oops I made a mistake", rather than "this is dumb; I wouldn't have had this issue in a proper programming language".

The main footguns are:

Quoting. Trust me you've got this wrong even with shellcheck. I have too. That's not a criticism. It's basically impossible to get quoting completely right in any vaguely complex Bash script.
Error handling. Sure you can set -e, but then that breaks pipelines and conditionals, and you end up with really monstrous pipelines full of pipefail noise. It's also extremely easy to forget set -e.
General robustness. Bash silently does the wrong thing a lot.

instead of a import os; os.args[1] in Python, you just do $1

No. If it's missing $1 will silently become an empty string. os.args[1] will throw an error. Much more robust.

Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.

Absolutely not. Python is strongly typed, and even statically typed if you want. Light years ahead of Bash's mess. Quoting is pretty easy to get right in Python.

I actually started keeping a list of bugs at work that were caused directly by people using Bash. I'll dig it out tomorrow and give you some real world examples.

[–] JamonBear 7 points 2 weeks ago (3 children)

Agreed.

Also gtfobins is a great resource in addition to shellcheck to try to make secure scripts.

For instance I felt upon a script like this recently:

#!/bin/bash
# ... some stuff ...
tar -caf archive.tar.bz2 "$@"

Quotes are OK, shellcheck is happy, but, according to gtfobins, you can abuse tar, so running the script like this: ./test.sh /dev/null --checkpoint=1 --checkpoint-action=exec=/bin/sh ends up spawning an interactive shell...

So you can add up binaries insanity on top of bash's mess.

load more comments (3 replies)

load more comments (7 replies)

[–] ShawiniganHandshake 23 points 2 weeks ago (2 children)

I've worked in bash. I've written tools in bash that ended up having a significant lifetime.

Personally, you lost me at

reading from the database

Database drivers exist for a reason. Shelling out to a database cli interface is full of potential pitfalls that don't exist in any language with a programmatic interface to the database. Dealing with query parameterization in bash sounds un-fun and that's table stakes, security-wise.

Same with making web API calls. Error handling in particular is going to require a lot of boilerplate code that you would get mostly for free in languages like Python or Ruby or Go, especially if there's an existing library that wraps the API you want to use in native language constructs.

load more comments (2 replies)

[–] [email protected] 17 points 2 weeks ago (2 children)

One thing that I don't think anyone else has mentioned is data structures. Bash does have arrays and hashmaps at least but I've found that working with them is significantly more awkward than in e.g. python. This is one of several reasons for why bash doesn't scale up well, but sure for small enough scripts it can be fine (if you don't care about windows)

[–] [email protected] 7 points 2 weeks ago (2 children)

I think I mentioned it, but inverse: The only data type I'm comfortable with in bash are simple string scalars; plus some simple integer handling I suppose. Once I have to think about stuff like "${foo[@]}" and the like I feel like I should've switched languages already.

Plus I rarely actually want arrays, it's way more likely I want something in the shape of

@dataclass(frozen=True)
class Foo:
    # …

foos: set[Foo] = …

load more comments (2 replies)

[–] [email protected] 3 points 2 weeks ago

That’s definitely worth mentioning indeed. Bash variables, aside from arrays and hashmaps that you get with declare, are just strings. Any time you need to start capturing a group of data and do stuff with them, it’s a sign to move on. But there are many many times where that’s unnecessary.

[–] [email protected] 14 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

We are not taking about use of Bash in dev vs use Bash in production. This is imho incorrect question that skirts around the real problem in software development. We talk about use of Bash for simple enough tasks where code is rarely changed ( if not written once and thrown away ) and where every primitive language or DSL is ok, where when it comes to building of medium or complex size software systems where decomposition, complex data structures support, unit tests, error handling, concurrency, etc is a big of a deal - Bash really sucks because it does not allow one to deal with scaling challenges, by scaling I mean where you need rapidly change huge code base according changes of requirements and still maintain good quality of entire code. Bash is just not designed for that.

[–] [email protected] 6 points 2 weeks ago (2 children)

But not everything needs to scale, at least, if you don’t buy into the doctrine that everything has to be designed and written to live forever. If robust, scalable solutions is the nature of your work and there’s nothing else that can exist, then yeah, Bash likely have no place in that world. If you need any kind of handling more complicated than just getting an error and doing something else, then Bash is not it.

Just because Bash isn’t designed for something you want to do, doesn’t mean it sucks. It’s just not the right tool. Just because you don’t practice law, doesn’t mean you suck; you just don’t do law. You can say that you suck at law though.

[–] [email protected] 9 points 2 weeks ago (1 children)

If your company ever has >2 people, it will become a problem.

load more comments (1 replies)

[–] [email protected] 5 points 2 weeks ago* (last edited 2 weeks ago)

Yep. Like said - "We talk about use of Bash for simple enough tasks ... where every primitive language or DSL is ok", so Bash does not suck in general and I myself use it a lot in proper domains, but I just do not use it for tasks / domains with complexity ( in all senses, including, but not limited to team work ) growing over time ...

[–] [email protected] 11 points 2 weeks ago* (last edited 2 weeks ago) (9 children)

A few responses for you:

I deeply despise bash (edit: this was hyperbole. I also deeply appreciate bash, as is appropriate for something that has made my life better for free!). That Linux shell defaults settled on it is an embarrassment to the entire open source community. (Edit: but Lexers and Parsers are hard! You don't see me fixing it, so yes, I'll give it a break. I still have to be discerning for production use, of course.)
Yes, Bash is good enough for production. It is the world's current default shell. As long as we avoid it's fancier features (which all suck for production use), a quick bash script is often the most reasonable choice.
For the love of all that is holy, put your own personal phone number and no one else's in the script, if you choose to use bash to access a datatbase. There's thousands of routine ways that database access can hiccup, and bash is suitable to help you diagnose approximately 0% of them.
If I found out a colleague had used bash for database access in a context that I would be expected to co-maintain, I would start by plotting their demise, and then talk myself down to having a severe conversation with them - after I changed it immediately to something else, in production, ignoring all change protocols. (Invoking emergency change protocols.)

Edit: I can't even respond to the security concerns aspect of this. Choice of security tool affects the quality of protection. In this unfortunate analogy, Bash is "the pull out method". Don't do that anywhere that it matters, or anywhere that one can be fired for security violations.

(Edit 2: Others have mentioned invoking SQL DB cleanup scripts from bash. I have no problem with that. Letting bash or cron tell the DB and a static bit of SQL to do their usual thing has been fine for me, as well. The nightmare scenario I was imagining was bash gathering various inputs to the SQL and then invoking them. I've had that pattern blow up in my face, and had a devil of a time putting together what went wrong. It also comes with security concerns, as bash is normally a completely trusted running environment, and database input often come from untrusted sources.)

[–] [email protected] 9 points 2 weeks ago (1 children)

Why internet man hate Bash? Bash do many thing. Make computer work.

[–] [email protected] 7 points 2 weeks ago* (last edited 2 weeks ago) (2 children)

I actually (also) love bash, and use it like crazy.

What I really hate is that bash is so locked in legacy that it's bad features (on a scripting language scale, which isn't fair) (and of which there are too many to enumerate) are now locked in permanently.

I also hate how convention has kept other shells from replacing bash's worst features with better modern alternatives.

To some extent, I'm railing against how hard it is to write a good Lexer and a Parser, honestly. Now that bash is stable, there's little interest in improving it. Particularly since one can just invoke a better scripting language for complex work.

I mourn the sweet spot that Perl occupies, that Bash and Python sit on either side of, looking longingly across the gap that separated their practical use cases.

I have lost hope that Python will achieve shell script levels of pragmatism. Although the invoke library is a frigging cool attempt.

But I hold on to my sorrow and anger that Bash hasn't bridged the gap, and never will, because whatever it can invoke, it's methods of responding to that invocation are trapped in messes like "if...fi".

load more comments (2 replies)

load more comments (8 replies)

[–] [email protected] 10 points 2 weeks ago (5 children)

At the level you're describing it's fine. Preferably use shellcheck and set -euo pipefail to make it more normal.

But once I have any of:

nested control structures, or
multiple functions, or
have to think about handling anything else than simple strings that other programs manipulate (including thinking about bash arrays or IFS), or
bash scoping,
producing my own formatted logs at different log levels,

I'm on to Python or something else. It's better to get off bash before you have to juggle complexity in it.

[–] [email protected] 3 points 2 weeks ago

If you're writing a lot of shell scripts and checking them with Shellcheck, and you're still convinced that it's totally safe... I tip my hat to you.

load more comments (4 replies)

[–] [email protected] 9 points 2 weeks ago

Over the last ten - fifteen years, I've written lots of scripts for production in bash. They've all served their purposes (after thorough testing) and not failed. Pretty sure one of my oldest (and biggest) is called temporary_fixes.sh and is still in use today. Another one (admittedly not in production) was partially responsible for getting me my current job, I guess because the interviewers wanted to see what kind of person would solve a coding challenge in bash.

However, I would generally agree that - while bash is good for many things and perhaps even "good enough" - any moderately complex problem is probably better solved using a different language.

[–] [email protected] 8 points 2 weeks ago (1 children)

I don't disagree with this, and honestly I would probably support just using bash like you said if I was in a team where this was suggested.

I think no matter how simple a task is there are always a few things people will eventually want to do with it:

Reproduce it locally
Run unit tests, integration tests, smoke tests, whatever tests
Expand it to do more complex things or make it more dynamic
Monitor it in tools like Datadog

If you have a whole project already written in Python, Go, Rust, Java, etc, then just writing more code in this project might be simpler, because all the tooling and methodology is already integrated. A script might not be so present for many developers who focus more on the code base, and as such out of sight out of mind sets in, and no one even knows about the script.

There is also the consideration that many people simply dislike bash since it's an odd language and many feel it's difficult to do simple things with it.

due to these reasons, although I would agree with making the script, I would also be inclined to have the script temporarily while another solution is being implemented.

load more comments (1 replies)

[–] [email protected] 7 points 2 weeks ago (1 children)

Wanna check if a variable’s set to something expected? if [[ <test goes here> ]]; then <handle>; fi

Hey, you can't just leave out "test goes here". That's worst part by a long shot.
The rest of the syntax, I will have to look up every time I try to write it, but at least I can mostly guess what it does when reading. The test syntax on the other hand is just impossible to read without looking it up.

I also don't actually know how to look that up for the double brackets, so that's fun. For the single bracket, it took me years to learn that that's actually a command and you can do man [ to view the documentation.

[–] [email protected] 4 points 2 weeks ago (1 children)

To be fair, you don’t always have to use the [[ syntax. I know I don’t, e.g. if I’m just looking for a command that returns 1 or 0, which happens quite a bit if you get to use grep.

That said, man test is my friend.

But I’ve also gotten so used to using it that I remember -z and -n by heart :P

[–] [email protected] 5 points 2 weeks ago (1 children)

If you need to use bash a lot just to learn 2 "keywords", then it's not a good language.

I have looked at bash scripts in the past, and even written some (small amount). I had to look up -z and -n every time. I've written a lot more python than bash, that's for sure. But even if I don't write python for a year, when needed I can just write an entire python script without minimal doc lookups. I just need to search if the function I want is part of syd, os or path.

The first time I want to do an else if my IDE will mark it red and I'll write eliffrom then on, same thing if I try to use { }.

If a bash script requires at least one array and one if statement, I can write the entire thing in python faster than I can search how to do those 2 things in bash.

load more comments (1 replies)

[–] 31337 7 points 2 weeks ago

It's ok for very small scripts that are easy to reason through. I've used it extensively in CI/CD, just because we were using Jenkins for that and it was the path of least resistance. I do not like the language though.

[–] [email protected] 6 points 2 weeks ago (1 children)

Bash is perfectly good for what you’re describing.

[–] [email protected] 3 points 2 weeks ago (1 children)

Serious question (as a bash complainer):

Have I missed an amazing bash library for secure database access that justifies a "perfectly good" here?

[–] [email protected] 3 points 2 weeks ago (2 children)

Every database I know comes with an SQL shell that takes commands from stdin and writes query results to stdout. Remember that "bash" never means bash alone, but all the command line tools from cut via jq to awk and beyond … so, that SQL shell would be what you call "bash library".

load more comments (2 replies)

[–] [email protected] 5 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

As I've matured in my career, I write more and more bash. It is absolutely appropriate for production in the right scenarios. Just make sure the people who might have to maintain it in the future won't come knocking down your door with torches and pitchforks...

load more comments (1 replies)

[–] [email protected] 4 points 2 weeks ago (4 children)

In your own description you added a bunch of considerations, requirements of following specific practices, having specific knowledge, and a ton of environmental requirements.

For simple scripts or duck tape schedules all of that is fine. For anything else, I would be at least mindful if not skeptical of bash being a good tool for the job.

Bash is installed on all linux systems. I would not be very concerned about some dependencies like sqlite, if that is what you're using. But very concerned about others, like jq, which is an additional tool and requirement where you or others will eventually struggle with diffuse dependencies or managing a managed environment.

Even if you query sqlite or whatever tool with the command line query tool, you have to be aware that getting a value like that into bash means you lose a lot of typing and structure information. That's fine if you get only one or very few values. But I would have strong aversions when it goes beyond that.

You seem to be familiar with Bash syntax. But others may not be. It's not a simple syntax to get into and intuitively understand without mistakes. There's too many alternatives of if-ing and comparing values. It ends up as magic. In your example, if you read code, you may guess that :- means fallback, but it's not necessarily obvious. And certainly not other magic flags and operators.

As an anecdote, I guess the most complex thing I have done with Bash was scripting a deployment and starting test-runs onto a distributed system (and I think collecting results? I don't remember). Bash was available and copying and starting processes via ssh was simple and robust enough. Notably, the scope and env requirements were very limited.

[–] [email protected] 6 points 2 weeks ago (1 children)

You seem to be familiar with Bash syntax. But others may not be.

If by this you mean that the Bash syntax for doing certain things is horrible and that it could be expressed more clearly in something else, then yes, I agree, otherwise I'm not sure this is a problem on the same level as others.

OP could pick any language and have the same problem. Except maybe Python, but even that strays into symbolic line noise once a project gets big enough.

Either way, comments can be helpful when strange constructs are used. There are comments in my own Bash scripts that say what a line is doing rather than just why precisely because of this.

But I think the main issue with Bash (and maybe other shells), is that it's parsed and run line by line. There's nothing like a full script syntax check before the script is run, which most other languages provide as a bare minimum.

load more comments (1 replies)

load more comments (3 replies)

[–] [email protected] 3 points 2 weeks ago (8 children)

May I introduce you to rust script? Basically a wrapper to run rust scripts right from the command line. They can access the rust stdlib, crates, and so on, plus do error handling and much more.

Anti Commercial-AI license

[–] [email protected] 2 points 2 weeks ago

Basically a wrapper to run rust scripts right from the command line.

Isn't that just Python? :v

[–] [email protected] 2 points 2 weeks ago (3 children)

Yeah, sometimes I'll use that just to have the sane control flow of Rust, while still performing most tasks via commands.

You can throw down a function like this to reduce the boilerplate for calling commands:

fn run(command: &str) {
    let status = Command::new("sh")
        .arg("-c")
        .arg(command)
        .status()
        .unwrap();
    assert!(status.success());
}

Then you can just write run("echo 'hello world' > test.txt"); to run your command.

load more comments (3 replies)

load more comments (6 replies)

[–] [email protected] 3 points 2 weeks ago

Dude, pihole is bash.

https://github.com/pi-hole/pi-hole/blob/master/pihole

[–] [email protected] 3 points 2 weeks ago

I've only ever used bash.

[–] [email protected] 2 points 2 weeks ago* (last edited 2 weeks ago) (6 children)

Run checkbashisms over your $PATH (grep for #!/bin/sh). That's the problem with Bash.
#!/bin/sh is for POSIX compliant shell scripts only, use #!/bin/bash if you use bash syntax.

Btw, i quite like yash.

load more comments (6 replies)

load more comments