Janet's Shenanigans

If you haven't read my previous blogposts on C++ coroutines, I would recommend doing so.

This post isn't an explanation of implementing something, as much as it is an explanation of something cool you can do in C++ using coroutines. Warning: this is very esoteric from a C++ perspective, and you're probably going to have problems understanding this post without knowing what algebraic effects are from other languages (mostly research ones), despite my best attempts to explain them.

In short, you can implement a rudimentary version of algebraic effects using C++ coroutines.

But what is an algebraic effect?

The simplest way to describe them is “resumable exceptions that can return a value”.

Koka explains it well with an example:

effect ask<a> {
  control ask() : a
}

This defines a generic effect ask<T> with an operation ask that returns a value of type T.

If a function wants to use this effect, it has to declare that it does, much in the same way that some languages require you to declare that your function can throw an exception.

fun ask-and-add-twice() : ask<int> int {
    return ask() + ask()
}

When you call this function, the Koka compiler checks that you handle ask somewhere in enclosing scopes, the same way that some languages check that your exceptions are being caught in a try/catch block.

Defining a handler for an effect in Koka looks like this:

fun ask-random() : random int {
  with control ask(){ resume(random-int()) }
  add-twice()
}

fun ask-const() : int {
  with control ask(){ resume(21) }
  add-twice()
}

So what does this have to do w/ C++ and coroutines?

C++ coroutines allow you to pause a function, and resume it, optionally passing in a value. This is the same thing that algebraic effects do: pause a function, and resume it, optionally passing in a value.

As a result, you can implement much of the functionality of algebraic effects in C++ using coroutines.

This is an example snippet of Crouton's rudimentary implementation of algebraic effects:

using Log = EffectVoidFun<std::function<void(QString)>>;

Effect<void> contextDependentLogging() {
    perform Log("hi");

    co_return;
}

void log() {
    {
        auto handler = Log::handler([](const QString& it) -> int {
            qDebug() << it;
        });

        contextDependentLogging();
    }
    {
        auto handler = Log::handler([](const QString& it) -> int {
            qWarning() << it;
        });

        contextDependentLogging();
    }
}

Algebraic effects are powerful in that they let your functions generic over behaviour, much in the same way that templates allow functions to be generic over types.

For example, say you're authoring a Matrix client. You can define an effect fetch that GETs/POSTs a URL. Without touching your code that uses fetch, you can transparently replace the handler for fetch in different parts of your application. Your live application could use a handler that uses the actual internet, while unit tests could use a handler that returns dummy data. Compared to #ifdef-ing different implementations, this allows different parts of an application to use different handlers at runtime, and allows library users to provide their own handlers without needing to modify the library.

There's a lot of stuff you can do with algebraic effects, and this only scratches the tip of the iceberg.

That's all for this blog post. Stay tuned for more C++-related shenanigans.

Contact Me

If you didn't understand anything here, please feel free to come to me and ask for clarification. This blog post is probably particularly confusing since it's deep in the realm of programming language geekery in the same way monads are.

Or, want to talk to me about other coroutine stuff I haven't discussed in these blog posts (or anything else you might want to talk about)?

Contact me here:

Telegram: https://t.me/pontaoski Matrix: #pontaoski:tchncs.de (prefer unencrypted DMs)

Tags: #libre

If you haven't read my previous blog post on coroutines in C++, you want to do that before reading this blog post.

In the last blog post, I explained how to construct an awaitable Future type.

If you've been tinkering with coroutines, you may have tried the following thing to allow awaiting on a pointer type:

auto operator await(QNetworkReply* it) {
    // ...
}

As far as the C++ compiler is concerned, this ain't kosher, because you're trying to define an operator for a primitive type (a pointer). Your compiler will probably tell you that this needs to be done on a class or enum type.

But that leaves the question of “how do I make a QNetworkReply* co_awaitable if I can't define co_await on a pointer type?” It is possible.

Your promise_type object has more jobs than what I covered in the last blog post. One of them is to potentially provide an await_transform function.

An await_transform function essentially “preprocesses” any values being co_awaited before the compiler attempts to look for a co_await implementation.

In code, if await_transform is defined on a promise_type, any co_await that looks like this:

auto response = co_await pendingValue;

will actually become:

auto response = co_await await_transform(pendingValue);

This is how we co_await a type you can't provide an operator co_await for: transform it into a type you can.

To integrate a QNetworkReply* into our coroutine types from before, this means we need to make an await_transform function that takes a QNetworkReply* and return a future. This is fairly simple.

auto await_transform(QNetworkReply* it) {
    auto future = Future();

    QObject::connect(it, &QNetworkReply::finished, it, [future, it]() {
        if (it->error() == QNetworkReply::NoError) {
            future.succeed(it);
        } else {
            future.fail(it);
        }
    });

    return future;
}

Make a future, mark as success if the reply succeeds, mark as failure if it doesn't succeed. Easy peasy.

Plonk this function in your promise_type from before.

This now allows you to do this:

Future fetch(QString url) {
    auto response = co_await nam.get(url);
    if (response.error() != QNetworkReply::NoError) {
        co_return response.readAll();
    }
    co_return false;
}

You can implement an await_transform for any input you like, and return anything you like, as long as you can co_await it.

The full code for this blog post can be found in a single-file format at https://invent.kde.org/-/snippets/1716. Compile with C++20 and -fcoroutines-ts -stdlib=libc++ for clang, and -fcoroutines for gcc.

A full library built on the style of coroutines introduced here (with type safe template variants) + other goodies for asynchronous Qt is available at https://invent.kde.org/cblack/croutons.

That's all for this blog post. Stay tuned for more C++-related shenanigans.

Contact Me

If you didn't understand anything here, please feel free to come to me and ask for clarification.

Or, want to talk to me about other coroutine stuff I haven't discussed in these blog posts (or anything else you might want to talk about)?

Contact me here:

Telegram: https://t.me/pontaoski Matrix: #pontaoski:tchncs.de (prefer unencrypted DMs)

Tags: #libre

Qt's networking code has always been one of its more obtuse parts, requiring using signals for something that didn't quite seem right for them. A linear flow of code would become a jumbled mess of member functions and signal connections.

When developing Challah's netcode, I quickly realised this wasn't going to suffice for the large amount of it I was going to be writing. Thus begins my journey through signal hell, arriving at many bumps before I discovered that integrating Qt's networking stuff with coroutines is possible.

Stop Zero

If you're just here to see some cool coroutine code, you can look at https://invent.kde.org/-/snippets/1711 for an single-file version of the tutorial in this blog-post, with a coroutine function already made and ready for you to play with. If you want something more full-fledged, look at https://invent.kde.org/cblack/coroutines.

But if you want to know what's actually going on behind the scenes, continue reading.

Stop One: Don't Do This

Approach one that I took to get out of signal hell was a simple while loop:

while (!reply->isFinished()) {
    QCoreApplication::processEvents();
}

This is a bad idea. Strange graphical glitches, bugs, crashes, etc. lie behind this innocuous code. Don't do it. Forcibly spinning the event loop in a lot of places causes a lot of bugs.

Step Two: Callback Hell

So, what do you do if you want to write somewhat linear code without doing that? Callbacks.

QObject::connect(val, &QNetworkReply::finished, [val, callback]() {
    if (val->error() != QNetworkReply::NoError) {
        val->deleteLater();
        callback({QStringLiteral("network failure(%1): %2").arg(val->error()).arg(val->errorString())});
        return;
    }
    
    auto response = val->readAll();
    
    protocol::chat::v1::CreateGuildResponse ret;
    if (!ret.ParseFromArray(response.constData(), response.length())) {
        val->deleteLater();
        callback({QStringLiteral("error parsing response into protobuf")});
        return;
    }
    
    val->deleteLater();
    callback({ret});
    return;
});

Every time I had to call into the netcode, I would have to provide a std::function<void(std::variant<Result,Error>)>. It's passable for a single call, but chaining them quickly violates “happy path sticks to the left”, making you wonder if you're actually writing Python with how far indented your code is.

c->call([c](auto response) {
  c->call([c](auto response) {
    c->call([c](auto response) {
      c->call([c](auto response) {
        c->call([c](auto response) {
        });
      });
    });
  });
});

Not good.

Step Three: Out of the Inferno, Into the Frying Pan

await/async as a mechanic in languages generally reduces the amount of callback hell you face by sugaring it for you. For example, in JS, this:

return fetch().then((it) => {
    return it.text
})

is equivalent to this:

return await fetch().text

Both of these return the exact same thing to the caller, but one is much easier to deal with when chained.

For a while, C++ didn't have this. Of course, with C++20, coroutines with co_await & company are now available* in most compilers.

Diversion: That Asterisk

Coroutines are technically available in most compilers, but you're going to have to do a lot of compiler-specific dances in both your code and the build system in order to pass the correct flags to enable coroutines.

I use this snippet to handle clang & gcc:

#if defined(__clang__)

#include <experimental/coroutine>
namespace ns = std::experimental;

#elif defined(__GNUC__)

#include <coroutine>
namespace ns = std;

#endif

where ns is aliased to the namespace containing coroutine-related stuff. Also note that clang will probably fail to compile stuff due to experimental/coroutine being part of libc++ while the rest of your system probably uses gcc's libstdc++. So, for all intents and purposes, gcc is your only option with coroutines on Linux.

Anyways...

Unhelpful Documentation

When hacking on coroutines, I quickly realised: the available documentation on what they were was both young and immature, as well as unhelpful in what they did have. So, this blog post is going to document them.

Parts of the Coroutine

Coroutines are largely split into two parts: the “promise” and the “awaitable”.

The promise is responsible for handling the “internal” side of any coroutine: it's what provides the return value, its yield behaviour, handling unhandled exceptions, the “await transformer”, and some other things. All coroutines need to return a type associated with one of these.

The awaitable handles the “external” side of the coroutine being coawaited. One of its first tasks is checking with the outside world whether or not the coroutine needs to suspend. If something is already available, there's no need to wait on it to become available. If the coroutine does need to wait, the awaitable receives a handle to reactivate it and is responsible for doing so, e.g. when an asynchronous network request completes and its results are available. The awaitable is also responsible for providing the final value returned to the user from coawaiting a coroutine.

As far as stuff that's directly interacting with coroutines goes, that's mostly it. However, there's still one type you need for an end-user API: the type returned by a coroutine, which is what the compiler uses to match your coroutine to a promise type.

SomeType a_coroutine() {
    auto foo = co_await another_thing();
    // ...
} 

This SomeType is what I will be calling the “future” or the “task” type, as it's basically the QFuture of the QPromise (or QFutureInterface to speak in Qt5 parlance).

C++ coroutines are very much “bring your own runtime and types”, and thankfully, that's what Qt already has, making it perfect for integrating into coroutines.

A QML-friendly future

C++ coroutines are the perfect fix to a long-standing issue in QML: representing asynchronous results in the UI in a manner that being an object property or model data can't.

So, let's get to constructing the base of our coroutines, and a type that will be suitable for passing into QML.

For ease of use, we'll want this done as a copyable Q_GADGET type with implicitly shared data. This will help us later down the road when we're passing the future type through lambdas.

(For the sake of simplicity of teaching how coroutines work, I'll be doing a simple non-template future type that talks in QVariants and doesn't have success/failure states. If you want to see code with more type safety and success/failure states, you can check out https://invent.kde.org/cblack/coroutines.)

We'll start by defining our shared data & behaviour.

class Future
{
    Q_GADGET

    struct Shared {
        QVariant result;
        std::function<void(QVariant)> then = [](QVariant) {};
        bool done = false;
    };
    QSharedPointer<Shared> d;

public:

    Future() {
        d.reset(new Shared);
    }
    Future(const Future& other) {
        this->d = other.d;
    }
};

This gives us a Future that is effectively a blob. First, let's define getters for done and result, so the outside world can actually tell what the current state of the Future is:

bool settled() const {
    return d->done;
}
QVariant result() const {
    return d->result;
}

Now we need a way to say that the future is completed and a result is available.

void succeed(const QVariant& value) const {
    if (d->done) {
        return;
    }
    d->done = true;
    d->result = value;
    d->then(d->result);
}

We don't want to trigger the callback again if we've already marked the future as done, so we abort early if we're already done.

You may notice that this function, despite being a setter semantically, is marked as const. This is mostly for ease of dealing with in lambdas later on.

Now, we need a way to register the callback function in the C++ side of things.

void then(std::function<void(QVariant)> then) const {
    d->then = then;

    if (d->done) {
        then(result());
    }
}

If the future is already done and you register a callback, you want to call it immediately. This is to handle situations like this:

Future fetch() {
  if (cache.has_thing()) {
    Future future;
    future.succeed(cache.get_thing();
    return future;
  }
  // ...
}
void main() {
    fetch().then([](QVariant) {
    });
}

If we didn't invoke the callback in then(), the callback would never be triggered as the future was returned to the caller in an already succeeded status.

Since I promised a QML-friendly future type, we should implement that now.

Q_INVOKABLE void then(const QJSValue& it) {
    then([va = it](QVariant result) mutable {
        va.call({va.engine()->toScriptValue(result)});
    });
}

We add a Q_INVOKABLE overload that takes a QJSValue, as JS functions are represented as QJSValues in C++. We then have a lambda which we pass back into the other then function, taking the QVariant, transforming it into a QJSValue, and calling the JavaScript callback with the variant.

JS usage looks like this:

future.then((it) => {
    console.warn(it)
})

If you've been following along, you should now have this class:

class Future
{

    Q_GADGET

    struct Shared {
        QVariant result;
        std::function<void(QVariant)> then = [](QVariant) {};
        bool done = false;
    };
    QSharedPointer<Shared> d;

public:

    Future() {
        d.reset(new Shared);
    }
    Future(const FutureBase& other) {
        this->d = other.d;
    }

    bool settled() const {
        return d->done;
    }
    QVariant result() const {
        return d->result;
    }

    void succeed(const QVariant& value) const {
        if (d->done) {
            return;
        }
        d->done = true;
        d->result = value;
        d->then(d->result);
    }

    void then(std::function<void(QVariant)> then) const {
        d->then = then;

        if (d->done) {
            then(result());
        }
    }
    Q_INVOKABLE void then(const QJSValue& it) {
        then([va = it](QVariant result) mutable {
            va.call({va.engine()->toScriptValue(result)});
        });
    }
};

The Backing Promise

Of course, the above class doesn't make a coroutine. For this, we need to implement the promise type.

For placement options, you have two places where you can put this: inside the future type itself, or outside of the type in an explicit coroutine_traits template specialisation.

The former would look like this:

class Future {
   // ...
   struct promise_type {
   };
};

While the latter would look like this:

template<typename ...Args>
struct ns::coroutine_traits<Future, Args...> {
    struct promise_type {
    };
};

(where ns is the namespace with the coroutine types, std:: on gcc or std::experimental on clang.)

The latter allows you to implement a promise type for any type, not just one that you have control of the declaration of.

For this blog post, I'll be going with the latter approach.

struct promise_type {
};

One of the things we'll need in our promise type is a member variable to hold the future type.

struct promise_type {
    Future _future;
};

Of course, the compiler doesn't know about this member variable, so we need to start implementing the promise_type interface:

struct promise_type {
    Future _future;
    Future get_return_object() noexcept {
        return _future;
    }
};

T get_return_object() noexcept is the exact type signature that needs to be implemented. We use noexcept here, as exceptions can cause a double free, and therefore, a crash.

Now, we need to implement two things: initial_suspend() and final_suspend(). These two functions are called at the start and end of your coroutine, and have to return a co_awaitable type.

In an expanded form, initialsuspend and finalsuspend look like this:

Future your_coroutine() {
    co_await promise.initial_suspend();
    // coroutine body here...
    co_await promise.final_suspend();
}

You could theoretically return anything you want here, but you'll likely be sticking with ns::suspend_never, which is an awaitable value that resumes execution of the coroutine immediately when co_awaited, and what we'll be using for this blog post.

Add the implementation to your promise type:

ns::suspend_never initial_suspend() const noexcept { return {}; }
ns::suspend_never final_suspend() const noexcept { return {}; }

Your promise is responsible for taking care of any values that are co_returned with the return_value function.

I recommend having two overloads: const T& (copy value) and T&& (move value).

For our promise_type, the implementation of those will simply look like this:

void return_value(const QVariant& value) noexcept
{
    _future.succeed(value);
}
void return_value(QVariant &&value) noexcept
{
    _future.succeed(std::move(value));
}

If your coroutine was returning something like a std::unique_ptr<T>, you would need the std::move in order to properly handle it, and thus why I say you should have this overload in your promise type.

With getreturnobject, initialsuspend, finalsuspend, and return_value, you only have one remaining function left to implement in your coroutine. It's quite simple.

void unhandled_exception() noexcept {

}

This is what gets called in the catch block of a try/catch block catching exceptions that the coroutine might throw. Use std::current_exception() to access the exception.

For this blog post, we'll simply be doing Q_ASSERT("unhandled exception"); in our implementation.

With that, the promise type is complete. It should look something like this:

struct promise_type {
    Future _future;

    Future get_return_object() noexcept {
        return _future;
    }

    ns::suspend_never initial_suspend() const noexcept { return {}; }
    ns::suspend_never final_suspend() const noexcept { return {}; }

    void return_value(const QVariant& value) noexcept
    {
        _future.succeed(value);
    }
    void return_value(QVariant &&value) noexcept
    {
        _future.succeed(std::move(value));
    }

    void unhandled_exception() noexcept {
        Q_ASSERT("unhandled exception");
    }
};

Remember that this needs to be within the Future type itself, or within the appropriate coroutine_traits overload.

The Awaitable

With the Future and promise completed, we don't have much left to do. We now need to implement operator co_await(Future) in order to allow the Future to be awaited.

Technically, we don't need the promise type to make a Future awaitable, only the Future and the co_await overload. Implementing a promise type allows your coroutine that's co_awaiting a Future to return another Future.

There's not really much to the co_await overload, so I'll just paste an implementation verbatim:

auto operator co_await(Future it) noexcept
{
    struct Awaiter {
        Future future;

        bool await_ready() const noexcept {
            return future.settled();
        }
        void await_suspend(ns::coroutine_handle<> cont) const {
            future.then([cont](QVariant) mutable {
                cont();
            });
        }
        QVariant await_resume() {
            return future.result();
        }
    };

    return Awaiter{ it };
}

co_await needs to return a type with the following methods:

bool await_ready() const noexcept: This function is called before suspending the coroutine to wait on the awaitable to resolve. If the function returns true, the coroutine continues execution without suspending. If the function returns false, it suspends and waits on the value.

Think back to the example of a function returning an already succeeded future to see why this is useful. If the future already holds a value, there's no need to waste time suspending the coroutine.

void await_suspend(ns::coroutine_handle<> callback) const: This function is called when the coroutine suspends with a coroutine handle passed in. For all intents and purposes, you can treat it as if it were a std::function holding a callback, except operator () resumes the coroutine instead of calling a function.

You call the coroutine_handle to resume the coroutine when whenever it's waiting on is ready, e.g. when the future in this example is marked as succeeded. You can connect this to just about anything, e.g. a QTimer::singleShot, a signal, using it as a callback, etc.

T await_resume() const: This is the value that gets assigned to the result of co_awaiting the expression:

auto it = co_await fetchFromNetwork();
  // |
  // ^ this is the value from await_resume()

Annd with all three types implemented, you now have a QML-friendly future, a promise, and an awaitable.

A Brand New Washing Machine

With your new coroutine, you can do just about anything asynchronously. This is an example function that returns a future that will be fufilled in N miliseconds:

Future timer(int duration) {
    Future it;

    QTimer::singleShot(duration, [it]() {
        it.succeed({});
    });

    return it;
}

Note that while it returns a Future, it is not a coroutine. It does not co_await. This is a completely normal function that you can call however you desire.

Making a coroutine using this function is easy:

Future thisIsACoroutine() {
    co_await timer(2000);

    co_return {};
}

Note that this function could not return void. The return type of a coroutine has to have an associated promise type, which void does not.

In a class exposed to QML...

class Singleton : public QObject
{
    Q_OBJECT

public:
    Q_INVOKABLE Future foo() {
        co_await timer(2000);

        co_return "i was returned by a coroutine";
    };
};

This can be used like so:

Button {
    anchors.centerIn: parent
    text: "a coroutine!"
    onClicked: Singleton.foo().then((val) => {
        console.warn(val)
    })
}

This will output the following in your console 2 seconds after every time you press this button:

qml: "i was returned by a coroutine"

The inline code from this blog post can be found in a single-file format at https://invent.kde.org/-/snippets/1711. Compile with C++20 and -fcoroutines-ts -stdlib=libc++ for clang, and -fcoroutines for gcc.

For a basic future+promise+awaitable deal, this is about all you need to know. However, the iceberg gets way deeper than that.

Stay tuned for more blog posts about the eldritch arcana you can pull off with coroutines. Next one will probably be about wrapping already-existing types such as a QNetworkReply* in order to make them co_await-able.

You can see more advanced (and arguably useful) code at https://invent.kde.org/cblack/coroutines.

Contact Me

Note that I wrote this blog post at 2 in the morning. If there's anything that doesn't make sense, please feel free to come to me and ask for clarification.

Or, want to talk to me about other coroutine stuff I haven't discussed in this blog post (or anything else you might want to talk about)?

Contact me here:

Telegram: https://t.me/pontaoski Matrix: #pontaoski:tchncs.de (prefer unencrypted DMs)

Tags: #libre

Tok's welcome screen

Bon venon! Herzlich willkommen! o kama pona! Tok now has an improved welcoming sequence with improved visuals and user experience.

Tok's password screen

If you want to see this for yourself, you can now log out using the new log out menu option.

Tok's log out menu

Once you're back in Tok, why not try out the new resizable chat list?

Tok's chat list

Or the toggleable sidebar?

Tok's sidebar

Photo and Video Sidebar

Tok now lets you view photos and videos that have been sent in a chat in the sidebar.

tok photo sidebar

tok video sidebar

Deletion

You might realise with the help of the sidebar that there's something inappropriate in your chat.

Now, Tok will allow you to properly delete others' messages if you have the correct permissions.

tok deletion

You can also now choose to only delete messages for yourself in groups/chats where this is applicable.

tok deletion dialog

Mentions

There's a good chance you may want to scold someone for sending an inappropriate photo in your chat. Tok now provides autocompletion for mentions, making it faster to type out someone's username.

tok mentioning

GIFs

Tok now supports displaying GIFs in chat.

Optimisations

Tok has seen a lot of optimisations. Memory usage should now hover around or below the 100MB range idle, depending on how many photos, videos, GIFs, etc. are on your screen.

Bugfixes

A lot of Tok's crashes have been fixed, especially ones pertaining to images.

Removed unneeded drums.

Obtaining Tok

Tok can be built from source from https://invent.kde.org/network/tok.

The support/development room is at https://t.me/kdetok.

Contributing

Interested in contributing? Come on by the dev chat and say hello.

Tags: #libre #tok

This week has seen a LOT of stuff come to Tok, both in terms of aesthetics and in functionality.

Tok now has support for displaying animated stickers.

Blur

tok with blur

Tok now offers an option to show the background behind its window with a blurry effect (on Plasma).

Music Player

tok showing music player

Tok now has an integrated music player allowing you to play music sent to you over Telegram. It integrates nicely with your desktop, showing up in the media controls.

Member List

tok showing member list

Tok now has a member list to the right of your chat, or by pressing the i button on mobile.

Improved File Handling

tok showing a file

Tok now has an improved file delegate, showing you an icon for the file, and you can now click the file to open it with an external program.

Colour Schemes

tok showing colour scheme selection

Tok now allows you to select colour schemes.

Minor Mobile Improvements

tok with mobile improvements

Tok has had a handful of improvements on mobile form factors, the most noticeable being the presence of the room picture when in the room view, and the back button correctly showing.

Bugfixes

Tok's stability has improved.

Removed unneeded drums.

Obtaining Tok

Tok can be built from source from https://invent.kde.org/network/tok.

The support/development room is at https://t.me/kdetok.

Contributing

Interested in contributing? Come on by the dev chat and say hello.

Tags: #libre #tok

that means it's time for the obligatory pride month blog post.

that means rainbows everywhere in your feed, and KDE is no different.

KDE has: – gay folks – bi folks – trans folks – enby folks – ace folks – a bunch of other fabulously queer folks

these folks are valid as heck and important to KDE (and a bunch of other FOSS orgs out there). much of our flagship software wouldn't be able to exist without the contributions of a ton of LGBT+ folks.

tags: #libre

Quick switcher

Tok is now equipped with a quick switcher for fast and efficient keyboard navigation between chats. Simply Ctrl+K, type in your chat's name, and press enter.

The search box above the chat list is now partially functional. It offers filtering of chats, though it does not offer the full search that Telegram is capable of offering.

Stickers

Sticker

Tok can now render non-animated stickers.

Videos

Tok now allows you to see videos in chat and open them in your preferred media player.

Join Messages

Join message being shown

Tok is now capable of displaying join messages.

Bugfixes

Removed unneeded drums.

Obtaining Tok

Tok can be built from source from https://invent.kde.org/network/tok.

The support/development room is at https://t.me/kdetok.

Contributing

Interested in contributing? Come on by the dev chat and say hello.

Tags: #libre #tok

Slim Mode

Tok now has a “slim mode”, for those who prefer a denser layout that follows IRC clients closer than it does mainstream messaging clients.

Compact mode

New Message Colours

The colours of messages have been adjusted to be more contrasting.

Keyboard Shortcuts

Tok now has alt-up and alt-down shortcuts for quickly switching between adjacent chats in the chat list.

Optimisation

Tok has seen some optimisation; switching between chats should now be much faster.

Bugfixes

Titlebar dragging in the MauiKit style now works as expected.

Obtaining Tok

Tok can be built from source from https://invent.kde.org/network/tok.

The support/development room is at https://t.me/kdetok.

Contributing

Interested in contributing? Come on by the dev chat and say hello.

Tags: #libre #tok

yay. you should be able to interact with me on the fediverse at @janet@blog.blackquill.cc. based?

While developing Challah (Qt chat app for Harmony protocol), I ran into situations where I needed to reference the same data in a reactive manner from multiple locations. In usual QtQuick fashion, you usually only have one one model per view, and views only ever get data from their assigned model or the surrounding context. This meant that in order to expose the same data in multiple locations, you had to wire signals between multiple models in order to synchronise their signal changes. This takes a lot of boilerplate to do and is extremely prone to error. I simply accepted this as something that you needed to do when using QtQuick.

My friend, Blusk, who is working on our web client caught this almost immediately when we were discussing how we implemented our clients with our respective UI frameworks (me QtQuick, him Vue.)

He was like “wtf, you don't have relational data/state in Qt? HOW” when I explained my struggles with the above problem.

I asked what relational state, and he gave a very long-winded explanation. In short, relational state is this:

using DataID = quint64;

struct Data {
    QString foo;
    QString bar;
}
struct Model {
    QList<DataID> data;
}
struct Store {
    QMap<DataID,Data> data;
}

instead of this:

struct Model {
    QList<Data> data;
}

This essentially involves splitting the Model into two parts: the Model and the Store. The Model is simply a list of IDs, while the Store maps IDs to data. Consumers can reactively listen to updates to the value of any key from the Store.

This technique is used to great effect in many JS frameworks, where you can simply bind to a map's key and any UI components will update as the map is updated.

This allows putting data into a single store and referring to it from multiple places.

Examples include:

  • Putting user data into a single store and referring to it from:

    • A list of users in a sidebar

    • Message delegates to display a message's author.

  • Message replies: simply look up the ID of the message being replied to in the Message store instead of doing model hacks where you expose a message's data in multiple places.

We lack an equivalent to that in Qt land, as we really only have reactive lists and reactive trees with the QAbstractItemModel hierarchy of classes.

Despite all the advantages of relational state, I didn't investigate using it for Challah because the code had already been written with the usual Qt list models & Harmony protocol is mostly fine w/out relational state.

That would have been the end of relational data and Qt for me.

And then I started Tok, a Kirigami Telegram client.

I started out writing Tok w/ the usual approach that I take for models and Qt, but then I quickly realised something: Telegram offers data in an aggressively relational manner.

Instead returning message data when you ask for chat history, Telegram simply returns a list of message IDs. Where do you get the message content? You receive a bunch of events asking you to place given messages in a local store. The events of Telegram's API can essentially be broken down into two types:

  • Events notifiying you of changed data, which should be placed in a local store
  • Events containing IDs pointing to the data that was given by the other type of event

This is basically Telegram telling you that you should use QList<ID> and QMap<ID,Data>. Problem is, while QList has a reactive sibling, the QAbstractListModel, the other half of the equation, the QMap, does not.

Enter QAbstractRelationalModel

I was like “hmm, this hole in capability needs to be fixed in Qt's models collection.”

However, due to Qt5 being dead to new features and Qt6 being a ways off to being usable for a KF/Kirigami app, I had to write something for Tok to use for now. So I did.

First things first, I added a README explaining why the classes were named as if they were official Qt types.

these are named as if they were Qt classes due to an intent to submit them to Qt for Qt6.

Simple enough, though I should probably clarify that I'm using Tok as a sort of testing ground for them.

Now for the code itself.

QAbstractRelationalModel is an extremely simple interface, designed to adapt the feel of the QAbstractItemModel to a key/value offering. Since QAbstractItemModel only works for lists and trees, I made QAbstractRelationalModel a subclass of QObject and not QAbstractItemModel.

Then, I defined methods for the API:

virtual QVariant data(const QVariant& key, int role = Qt::DisplayRole) const = 0;

This is QAbstractRelationalModel's equivalent to the QAbstractItemModel's equivalently data function. Instead of taking a parent/row/column tuple (QModelIndex), QAbstractRelationalModel takes a QVariant in order to represent many types of keys with the same interface.

virtual bool checkKey(const QVariant& key) const = 0;

This is the equivalent of checkIndex from QAbstractItemModel. Give it a key, and it tells you if that key present in the model.

virtual bool canFetchKey(const QVariant& key);
virtual bool fetchKey(const QVariant& key);

These are the equivalent of canFetchMore and fetchMore from the QAbstractItemModel. However, instead of simply being for appending data to the model, these methods let you query whether or not any given key can be fetched.

virtual QHash<int, QByteArray> roleNames();

This shouldn't need any explanation. This works exactly how it does in QAbstractItemModel.

That leaves us with the data reactivity part. In Qt, data reactivity is done through signals. A set of three signals is enough to suffice for our needs:

void keyAdded(const QVariant& key);
void keyRemoved(const QVariant& key);
void keyDataChanged(const QVariant& key, const QVector<int>& roles);

These are called to notify you about Create/Update/Delete changes to data in the model.

Usage: QQmlRelationalListener

Now we need something to use it. Since Tok is a QtQuick application, I wrote a QML component that allows listening to a key provided by a QAbstractRelationalModel.

Usage is fairly simple and looks like this:

RelationalListener {
    id: messageData
    key: delegate.messageID
    shape: QtObject {
        required property string messageContent
        required property string messageAuthorID
    }
}
QQC2.Label {
    text: messageData.data.messageContent
}

Easy peasy.

The implementation of this listener isn't really as interesting as the model itself, so I won't go too into depth here. You can look at its source on invent.kde.org.

The most interesting thing here is probably the shape property. It's essentially the component that the listener instantiates and utilises to expose data to the user.

This takes advantage of the “new” required property syntax in Qt 5.15, which allows Qt to loudly abort the application when a programming error is made instead of silently yielding pesky undefineds.

You may also be confused as to how I said that was a component, as to the user, it looks like instantiating a QtObject in QML. Simply enough, the QML engine allows you to use T {} instead of Component { T { } } for properties of the QQmlComponent* type.

Usage in Tok

I spent most of today and yesterday porting Tok from plain QAbstractItemModels to QAbstractItemModels + QAbstractRelationalModels. This basically meant porting the messages and the user data to a model/store architecture. I quickly noticed that my code felt much more elegant: changes to data were entirely separated from changes to different views on that data, e.g. changing user data used by both the messages view and a user list view, or changing message data used both by a message replying to it and the message itself. The models themselves also were reduced in complexity: messages model only has to worry about ordering and adding/removing IDs from the list as Telegram dictates, and the messages store only has to worry about adding and notifiying about updates from Telegram. No more “when message changed, try to locate the child model for the chat that that message belongs to and post the event there if it's present, otherwise don't.” It's simply “when message received, change the message store.”

Conclusion

After using relational data, I strongly agree with my friend's bewildered reaction to me saying there was nothing like relational state in Qt. Many things that used to be “how do” or mistake-prone boilerplate hells for me now have readily apparent solutions for me now.

Contact Me

If you have any thoughts on this post, feel free to share them w/ me in #chat:kde.org or https://t.me/kdechat, or by DMing me at @pontaoski on Telegram or @pontaoski:kde.org on Matrix.