liori

I'm working on a query engine, essentially a tool to scan/filter/annotate by lookups/group by/aggregate a large dataset, tens-of-terabytes range. The compute part seems to be a bottleneck for me (I'll be doing around 80-300 GB/s of reads, and yes, I will have hardware capable of providing that kind of throughput). My hypothesis is that by encoding query in form of template arguments I can make the compiler generate code optimized for a specific type of query (like, the filtering or aggregation keys). But I do not know what queries will users send, so I need a way to instantiate templates at runtime.

Sounds simple: for a new type of query invoke a compiler at runtime to build a dynamic library with a new instantiation, then dynload it and off we go. Some prior work is here, though I'm pretty sure any JIT compiler also can counts here. But there's enough technical details to worry about, and at the same time this idea isn't novel, so I wonder—are there any packaged solutions for this kind of approach?

TL;DR: we have discovered XMPP (Jabber) instant messaging protocol encrypted TLS connection wiretapping (Man-in-the-Middle attack) of jabber.ru (aka xmpp.ru) service’s servers on Hetzner and Linode hosting providers in Germany. The attacker has issued several new TLS certificates using Let’s Encrypt service which were used to hijack encrypted STARTTLS connections on port 5222 using transparent MiTM proxy. The attack was discovered due to expiration of one of the MiTM certificates, which haven’t been reissued. There are no indications of the server breach or spoofing attacks on the network segment, quite the contrary: the traffic redirection has been configured on the hosting provider network. The wiretapping may have lasted for up to 6 months overall (90 days confirmed). We believe this is lawful interception Hetzner and Linode were forced to setup.

In short, by the exit polls:

• PiS (the current ruling party, right-wing) got the most votes, but cannot rule alone, nor can rule in coalition with the alt-right party Konfederacja by a decent margin (212 mandates total vs. 231 needed to have majority).
• The opposition parties (center-right Koalicja Obywatelska, center-right Trzecia Droga and leftist Lewica) together have majority. They know they have to make a government together, the question is whether they can overcome their differences. They did suggest strong coöperation during their campaigns.
• Highest-ever turnout (72%) in the elections.
• The accompanying referendum (a device to have more funds for promoting ideas by the current ruling party) a total failure (40% turnout—voters had to explicitly opt-out of participation!).

While high-frequency trading is not exactly my favourite topic, I do like reading on their technical approaches.

By Paul Bilokon, Burak Gunduz

This work aims to bridge the existing knowledge gap in the optimisation of latency-critical code, specifically focusing on high-frequency trading (HFT) systems. The research culminates in three main contributions: the creation of a Low-Latency Programming Repository, the optimisation of a market-neutral statistical arbitrage pairs trading strategy, and the implementation of the Disruptor pattern in C++. The repository serves as a practical guide and is enriched with rigorous statistical benchmarking, while the trading strategy optimisation led to substantial improvements in speed and profitability. The Disruptor pattern showcased significant performance enhancement over traditional queuing methods. Evaluation metrics include speed, cache utilisation, and statistical significance, among others. Techniques like Cache Warming and Constexpr showed the most significant gains in latency reduction. Future directions involve expanding the repository, testing the optimised trading algorithm in a live trading environment, and integrating the Disruptor pattern with the trading algorithm for comprehensive system benchmarking. The work is oriented towards academics and industry practitioners seeking to improve performance in latency-sensitive applications.

I've said this previously, and I'll say it again: we're severely under-resourced. Not just XFS, the whole fsdevel community. As a developer and later a maintainer, I've learnt the hard way that there is a very large amount of non-coding work is necessary to build a good filesystem. There's enough not-really-coding work for several people. Instead, we lean hard on maintainers to do all that work. That might've worked acceptably for the first 20 years, but it doesn't now.

[…]

Dave and I are both burned out. I'm not sure Dave ever got past the 2017 burnout that lead to his resignation. Remarkably, he's still around. Is this (extended burnout) where I want to be in 2024? 2030? Hell no.

We are working on a tool that essentially allows external customers to access various extracts of our datasets, with parameterized filtering, aggregation, the usual stuff, though REST API. Some of these extracts are time consuming to prepare, so we are looking for ways to manage asynchronous report generation or making it possible for customers to schedule reports upfront, as opposed to having a synchronous API. There are tons of libraries for implementing synchronous REST APIs, but are there any standard approaches or tools for this kind of asynchronous cross-organizational communication? Like, maybe something that would allow each customer inspect their schedules and pending queries, configure how they want the results to be delivered? I fear we will need to build something like that from scratch.

Some time ago I was looking for a sources that would give me in-depth understanding of performance characteristics of large-scale storage. This is the best text on hard disk drives I've found so far, explaining details such as various switch times, zoned recording or head skew. It's almost 20 years old, though, and so misses some developments. I wonder if you know any more modern sources?