Ask /r/HomeNetworking
Maybe something like OpenWRT or Mikrotiks RouterOS can run on it.
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
For Example
We welcome posts that include suggestions for good self-hosted alternatives to popular online services, how they are better, or how they give back control of your data. Also include hints and tips for less technical readers.
Useful Lists
Ask /r/HomeNetworking
Maybe something like OpenWRT or Mikrotiks RouterOS can run on it.
None.
It's exceedingly unlikely that ANY general-use network OS will recognize or even run on it. Even if it did, as it lacks actual switching hardware, it would make a very poor switch.
As for "change its ... VLAN [configuration] with a simple click" that's almost certainly an XY problem.
Most of these boxes are just Celeron or Intel i3 based computers with a few extra NICs, they'll run pretty much anything just fine. They don't have actual switching hardware, but they've got enough CPU juice to push gigabit connections with a few simple rules/filters. I have an n5105 based one with six i226 NICs, with a simple bridge it can throughput close to 2.5gbps in iperf.
OPNsense will do what you want, although it's probably overkill in this case. Also keep in mind that any switching it does will be purely software, so it might struggle to push heavy traffic through, especially if you add rules and vlans and such. Openwrt is more lightweight and should also work, or you could just install a general Linux OS like Ubuntu or debian and use iptables to route stuff. The better option though is to just get a proper switch, since those have hardware switching.