this post was submitted on 24 Dec 2023
204 points (95.5% liked)

Linux

48363 readers
1391 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
204
submitted 11 months ago* (last edited 11 months ago) by [email protected] to c/[email protected]
 

I've been reading Mastering Regular Expressions by Jeffrey E.F. Friedl, and since nobody in my life (aside from my wife) cares, I thought I’d share something I'm pretty proud of. My first set of regular expressions, that I wrote myself to manipulate the text I'm working with.

What’s I’m so happy about is that I wrote these expressions. I understand exactly what they do and the purpose of each character in each expression.

I've used regex in the past. Stuff cobbled together from stack overflow, but I never really understood how they worked or what the expressions meant, just that they did what I needed them to do at the time.

I'm only about 10% of the way through the book, but already I understand so much more than I ever did about regex (I also recognize I have a lot to learn).

I wrote the expressions to be used with egrep and sed to generate and clean up a list of filenames pulled out of tarballs. (movies I've ripped from my DVD collection and tarballed to archive them).

The first expression I wrote was this one used with tar and egrep to list the files in the tarball and get just the name of the video file:

tar -tzvf file.tar.gz | egrep -o '\/[^/]*\.m(kv|p4)' > movielist

Which gives me a list of movies of which this is an example:

/The.Hunger.Games.(2012).[tmdbid-70160].mp4

Then I used sed with the expression groups to remove:

  • the leading forward slash
  • Everything from .[ to the end
  • All of the periods in between words

And the last expression checks for one or more spaces and replaces them with a single space.

This is the full sed command:

sed -Eie 's/^\///; s/\.\[[a-z]+-[0-9]+\]\.m(p4|kv)//; s/[^a-zA-Z0-9\(\)&-]/ /g; s/ +/ /g' movielist

Which leaves me with a pretty list of movies that looks like this:

The Hunger Games (2012)

I'm sure this could be done more elegantly, and I'm happy for any feedback on how to do that! For now, I'm just excited that I'm beginning to understand regex and how to use it!

Edit: fixed title so it didn’t say “regex expressions”

top 50 comments
sorted by: hot top controversial new old
[–] leo 117 points 11 months ago (2 children)

Knowledge and understanding. Feels good, man.

Obligatory Xkcd.

[–] [email protected] 26 points 11 months ago

It does feel good! And thanks for that xkcd! That one’s new to me.

[–] [email protected] 21 points 11 months ago (1 children)

Ah....the days when perl was the shit and python was still a glimmer in the eye of some frustrated programmer.

[–] [email protected] 67 points 11 months ago (1 children)

I relearn regex from scratch every time I need to use it.

[–] [email protected] 6 points 11 months ago

This is the way.

[–] [email protected] 37 points 11 months ago* (last edited 11 months ago) (5 children)

Good job !

I highly recommend trying out the various online regex editor.

These WISIWIG kind of editors are great because you immediately see what the regex is catching and for what reason.

I took the first one in my search results but try different ones.

https://regex101.com/

Also I used GPT to get some regex for some specific strings and it can be helpful to get a quickstart at building a specific regex.

In that case I was building a regex for a specific log from postfix.

PS: just make sure to select the correct flavor of regex you are using in these online tools.

Edit: Also one of my favorite YT channels has pretty cool videos on RegEx : https://youtu.be/6gddK-cOxYc?si=0bnNkSDzifjdxwjU

[–] [email protected] 10 points 11 months ago (2 children)

Wait. Are there flavors of regex? Every time I have to use regex it hurts my brain and I never need to do it enough to actually sit down and learn it properly like OP is doing. Just knowing there are different ways of doing the same things in an already mind baffeling language blows me away even more.

[–] [email protected] 20 points 11 months ago* (last edited 11 months ago) (3 children)

Yeah. The only one you really need to care about (especially under Linux) is PCRE, the good 'ol Perl Compatible Regular Expressions. For the most part, every other flavor is a derivative of that. Microsoft had a weird version for a while, but that may be completely dead now, thankfully.

Learning the syntax of regex is fairly easy. Hell, I still have to use this cheat sheet more often now that my perl skills are no longer needed or even relevant.

Regex isn't that hard. The challenge is identifying and understanding patterns in the data that you are filtering. Here is a brain hack: As an example, if to have pages and pages of logs that you need to filter, open up one of the log files, stare at the screen and hold the page down key for several dozen pages. Patterns can be easily seen in the blur of text that is quickly scrolling across the screen. (Our brains love to find patterns in noise, btw.) The patterns that you see will give you focus points for developing regular expressions to match. ie: You start breaking strings into chunks and seeing the ebb and flow of data streaming across a screen helps. Anomalies in the data "stream" are are easy to spot as well.

From a security and efficiency standpoint, you should also understand where the most processing takes place so you don't kill whatever platform you are working on.

Sorry for the rambling, but I am getting older and feel the need to pass on a ton of tips and tricks whenever I can for these "archaic" languages.

[–] [email protected] 6 points 11 months ago

That screen scrolling tip is gold. I’ve often used that trick to spot anomalies in data. Hadn’t considered using it to spot the patterns for regex.

[–] [email protected] 2 points 11 months ago* (last edited 11 months ago) (2 children)

The only one you really need to care about (especially under Linux) is PCRE,

Well, no. sed, grep, awk, vi etc. use POSIX regexes. GNU implementations also provide perl compatible mode via an unportable option. In modern programming languages like go and rust standard regex engines are compatible to RE2 - relatively new dialect developed in Google that is not described in the Friedl's book (you may think of it as an extension of extended POSIX dialect). Even raku has its own dialect incompatible to perl as well as other ones.

Nowadays it is common to move away from perl-like engines, however they are still widely used in PCRE based software and software written in python, JS etc.

load more comments (2 replies)
load more comments (1 replies)
[–] ricecake 5 points 11 months ago

Yes. Most things use pcre, or Perl Compatible Regular Expressions, but there are other flavors. Usually they lack features or have slightly different syntax.

[–] [email protected] 6 points 11 months ago

Regex101 is amazing. It tends to balk at backtracing which we rely on a lot for work, but it's such a good visual.

Chat GPT can also save a lot of time writing regex, but it tends to write very unreadable regex because it thinks it's being clever when it really isnt.

Regex is an art form, and writing readable regex is another step above that.

[–] [email protected] 2 points 11 months ago* (last edited 11 months ago)

Thank you very much. I will definitely check out the regex builders. That’ll be super useful

Edit: fix stupid autocorrect turning regex into Reyes.

[–] [email protected] 2 points 11 months ago

Computerphile! I’ll check those out.

[–] [email protected] 2 points 11 months ago

Piggybacking onto this to mention my go-to online RegEx editor: RegExr. It lets you test the regex as you type, explains the particular symbols used, as well as has a sidebar where you can see different pattern types categorically. I've been using it for almost 2 years now, and haven't had any reason to use much else (after I discovered this).

[–] [email protected] 14 points 11 months ago* (last edited 11 months ago) (1 children)

It is a great book, although a bit outdated. In particular, nowadays egrep is not recommended to use. grep -E is a more portable synonim.

Some notes on you script:

  1. You don't need to escape slashes in grep regex. In the sed s/// command better use another character like s### so you also can leave slashes unescaped.

  2. You usually don't need to pipe grep and sed, sed -n with regex address and explicit printing command gives the same result as grep.

  3. You could omit leading slash in your egrep regex, so you won't need to remove it later.

So I would do the same with

tar -tzvf file.tar.gz | sed -En '/\.(mp4|mkv)$/{s#^.*/##; s#\.\[.*##; s#[^a-zA-Z0-9()&-]# #g; s/ +/ /g; p}'
[–] [email protected] 4 points 11 months ago (6 children)

nowadays egrep is not recommended to use. grep -E is a more portable synonim

Not directed at you personally, but this is the kind of pointless pedantry from upstream developers that grinds my gears.

Like, I've used egrep for 25 years. I don't know of a still relevant Unix variant in existence that doesn't have the egrep command. But suddenly now, when any other Unix variant but Linux is all but extinct, and all your shell scripts are probably full of bashisms and Linuxisms anyway, now there is somehow a portability problem, and they deem it necessary to print out a warning whenever I dare to run egrep instead of grep -E? C'mon now ... If anything, they have just made it less portable by spitting out spurious warnings where there weren't any before.

load more comments (6 replies)
[–] [email protected] 12 points 11 months ago (1 children)

Just to chip in because I haven't seen it mentioned yet, but I fing LLMs like ChatGPT or Microsoft Copilot are really good at making regexes and also at explaining regexes. So if you're learning them or just want to get the darned thing to work so you can go to bed those are a good resource.

[–] [email protected] 4 points 11 months ago (1 children)

You know, I haven’t yet used ChatGPT for anything, I might check it out for this reason.

[–] [email protected] 4 points 11 months ago

I use it to tell me which page of the Pathfinder 1e manual I should look on for the rules I need.

[–] [email protected] 12 points 11 months ago (1 children)

Just adding my congrats. Good job, OP. Regex is super useful stuff.

[–] [email protected] 2 points 11 months ago

Thank you!!!

[–] [email protected] 11 points 11 months ago (1 children)

Nice! Learning regular expressions is one of those things where it’s absurd but once you do it, you can solve problems that bedevil whole industries.

[–] [email protected] 6 points 11 months ago

Thanks!

And it still kinda breaks my brain when I look at an expression. When I just look at it it looks like utter gibberish, but when I say to myself, “okay, what’s this doing?”

And go through it character by character, it turns into something I can comprehend.

[–] [email protected] 10 points 11 months ago (1 children)

"regex" means "regular expression", so "regex expression" means "regular expression expression".

[–] [email protected] 4 points 11 months ago

Dang! I read through my post three times to make sure I didn’t do that and completely missed that I did it right in the title. (Now fixed).

[–] [email protected] 8 points 11 months ago (2 children)

I’ll have to check out this book. Just remember HTML cannot be parsed with regex

[–] [email protected] 3 points 11 months ago* (last edited 11 months ago)

Well, technically it is possible with regex dialect that has lookarounds, but it is overcomplicated. There's really no reason to do it.

[–] [email protected] 2 points 11 months ago

Thanks for that link.

[–] [email protected] 7 points 11 months ago (2 children)

I think the most impressive part of this is that your wife cares.

...does she have a sister?

[–] [email protected] 3 points 11 months ago (1 children)

I'm currently seeing a girl I started dating after she had problems with her regex and I helped her out.

So far so good.

load more comments (1 replies)
load more comments (1 replies)
[–] [email protected] 6 points 11 months ago (4 children)
load more comments (4 replies)
[–] [email protected] 5 points 11 months ago* (last edited 11 months ago)

I can also recommend the book the TS mentioned, it is very good and after reading it you will understand regular expressions. It's fine to use a cheat sheet if you want, cause if you don't do it regularly the knowledge can sag, but the understanding is what matters. Also depending on the context, different implementations can have slightly different syntax or modifiers to be aware of.

I lent out the book to my brother once and he somehow lost it, so I never got it back. Don't lend out book guys.

And remember not everything can be solved using a regular expression: https://xkcd.com/1171/

[–] [email protected] 5 points 11 months ago* (last edited 11 months ago) (1 children)

Give a man a regular expression and he’ll match a string… teach him to make his own regular expressions and you’ve got a man with problems. -- yakugo in http://regex.info/blog/2006-09-15/247#comment-3022 (and yes, it is http:// never https:// for this domain)

load more comments (1 replies)
[–] [email protected] 5 points 11 months ago

I stumbled upon this regex crossword puzzle a while back. I was never good enough to get it, but it seems like it could be fun.

[–] [email protected] 4 points 11 months ago (1 children)

That's cool! Kudos!

My biggest project was to remove leading and trailing whitespaces but I think I failed twice 😅

[–] [email protected] 2 points 11 months ago (1 children)

🤣

I went though about 20 iterations to get all of this to work correctly.

[–] [email protected] 7 points 11 months ago (1 children)

Why spend 20 minutes manually changing text in a file, when you can spend 90 minutes figuring out a single RegEx to do it?

load more comments (1 replies)
[–] [email protected] 3 points 11 months ago (1 children)

That's really cool! I know some regex and I tried to learn vim regex, only to find out it's a rabbithole so deep I'm afraid to look into. The feeling when you press enter and your carefully crafted regex does exactly what it's supposed to do is awesome though. Good luck!

[–] [email protected] 2 points 11 months ago (1 children)

Vim is on my list of things to learn. I didn’t even know vim had its own regex, but I suppose that makes sense. I’ve messed with vim a bit, but have stuck to nano so far.

load more comments (1 replies)
[–] [email protected] 3 points 11 months ago (1 children)

Regexps are awesome! And also not at the same time:-P. 🎉 Congrats👏!:-)

load more comments (1 replies)
[–] [email protected] 3 points 11 months ago (1 children)

this is definitely something to be proud of. great work, keep it up!

[–] [email protected] 2 points 11 months ago

Thank you! I plan to, this has given me more motivation!

[–] [email protected] 3 points 11 months ago

Congrats on your learning! I did a similar thing with music and converting all random songs to mp3

[–] [email protected] 3 points 11 months ago

I was wondering a few years ago how far you could get with implementing some simple markup syntax with just regex. Turns out, surprisingly far, but once stuff starts going wrong you're in a less than ideal situation.

https://github.com/bwachter/awfulcms/blob/master/lib/AwfulCMS/SynBasic.pm

load more comments
view more: next ›