Janet's Shenanigans

While developing Challah (Qt chat app for Harmony protocol), I ran into situations where I needed to reference the same data in a reactive manner from multiple locations. In usual QtQuick fashion, you usually only have one one model per view, and views only ever get data from their assigned model or the surrounding context. This meant that in order to expose the same data in multiple locations, you had to wire signals between multiple models in order to synchronise their signal changes. This takes a lot of boilerplate to do and is extremely prone to error. I simply accepted this as something that you needed to do when using QtQuick.

My friend, Blusk, who is working on our web client caught this almost immediately when we were discussing how we implemented our clients with our respective UI frameworks (me QtQuick, him Vue.)

He was like “wtf, you don't have relational data/state in Qt? HOW” when I explained my struggles with the above problem.

I asked what relational state, and he gave a very long-winded explanation. In short, relational state is this:

using DataID = quint64;

struct Data {
    QString foo;
    QString bar;
}
struct Model {
    QList<DataID> data;
}
struct Store {
    QMap<DataID,Data> data;
}

instead of this:

struct Model {
    QList<Data> data;
}

This essentially involves splitting the Model into two parts: the Model and the Store. The Model is simply a list of IDs, while the Store maps IDs to data. Consumers can reactively listen to updates to the value of any key from the Store.

This technique is used to great effect in many JS frameworks, where you can simply bind to a map's key and any UI components will update as the map is updated.

This allows putting data into a single store and referring to it from multiple places.

Examples include:

• Putting user data into a single store and referring to it from:

  • A list of users in a sidebar

  • Message delegates to display a message's author.

• Message replies: simply look up the ID of the message being replied to in the Message store instead of doing model hacks where you expose a message's data in multiple places.

We lack an equivalent to that in Qt land, as we really only have reactive lists and reactive trees with the QAbstractItemModel hierarchy of classes.

Despite all the advantages of relational state, I didn't investigate using it for Challah because the code had already been written with the usual Qt list models & Harmony protocol is mostly fine w/out relational state.

That would have been the end of relational data and Qt for me.

And then I started Tok, a Kirigami Telegram client.

I started out writing Tok w/ the usual approach that I take for models and Qt, but then I quickly realised something: Telegram offers data in an aggressively relational manner.

Instead returning message data when you ask for chat history, Telegram simply returns a list of message IDs. Where do you get the message content? You receive a bunch of events asking you to place given messages in a local store. The events of Telegram's API can essentially be broken down into two types:

• Events notifiying you of changed data, which should be placed in a local store
• Events containing IDs pointing to the data that was given by the other type of event

This is basically Telegram telling you that you should use QList<ID> and QMap<ID,Data>. Problem is, while QList has a reactive sibling, the QAbstractListModel, the other half of the equation, the QMap, does not.

Enter QAbstractRelationalModel

I was like “hmm, this hole in capability needs to be fixed in Qt's models collection.”

However, due to Qt5 being dead to new features and Qt6 being a ways off to being usable for a KF/Kirigami app, I had to write something for Tok to use for now. So I did.

First things first, I added a README explaining why the classes were named as if they were official Qt types.

these are named as if they were Qt classes due to an intent to submit them to Qt for Qt6.

Simple enough, though I should probably clarify that I'm using Tok as a sort of testing ground for them.

Now for the code itself.

QAbstractRelationalModel is an extremely simple interface, designed to adapt the feel of the QAbstractItemModel to a key/value offering. Since QAbstractItemModel only works for lists and trees, I made QAbstractRelationalModel a subclass of QObject and not QAbstractItemModel.

Then, I defined methods for the API:

virtual QVariant data(const QVariant& key, int role = Qt::DisplayRole) const = 0;

This is QAbstractRelationalModel's equivalent to the QAbstractItemModel's equivalently data function. Instead of taking a parent/row/column tuple (QModelIndex), QAbstractRelationalModel takes a QVariant in order to represent many types of keys with the same interface.

virtual bool checkKey(const QVariant& key) const = 0;

This is the equivalent of checkIndex from QAbstractItemModel. Give it a key, and it tells you if that key present in the model.

virtual bool canFetchKey(const QVariant& key);
virtual bool fetchKey(const QVariant& key);

These are the equivalent of canFetchMore and fetchMore from the QAbstractItemModel. However, instead of simply being for appending data to the model, these methods let you query whether or not any given key can be fetched.

virtual QHash<int, QByteArray> roleNames();

This shouldn't need any explanation. This works exactly how it does in QAbstractItemModel.

That leaves us with the data reactivity part. In Qt, data reactivity is done through signals. A set of three signals is enough to suffice for our needs:

void keyAdded(const QVariant& key);
void keyRemoved(const QVariant& key);
void keyDataChanged(const QVariant& key, const QVector<int>& roles);

These are called to notify you about Create/Update/Delete changes to data in the model.

Usage: QQmlRelationalListener

Now we need something to use it. Since Tok is a QtQuick application, I wrote a QML component that allows listening to a key provided by a QAbstractRelationalModel.

Usage is fairly simple and looks like this:

RelationalListener {
    id: messageData
    key: delegate.messageID
    shape: QtObject {
        required property string messageContent
        required property string messageAuthorID
    }
}
QQC2.Label {
    text: messageData.data.messageContent
}

Easy peasy.

The implementation of this listener isn't really as interesting as the model itself, so I won't go too into depth here. You can look at its source on invent.kde.org.

The most interesting thing here is probably the shape property. It's essentially the component that the listener instantiates and utilises to expose data to the user.

This takes advantage of the “new” required property syntax in Qt 5.15, which allows Qt to loudly abort the application when a programming error is made instead of silently yielding pesky undefineds.

You may also be confused as to how I said that was a component, as to the user, it looks like instantiating a QtObject in QML. Simply enough, the QML engine allows you to use T {} instead of Component { T { } } for properties of the QQmlComponent* type.

Usage in Tok

I spent most of today and yesterday porting Tok from plain QAbstractItemModels to QAbstractItemModels + QAbstractRelationalModels. This basically meant porting the messages and the user data to a model/store architecture. I quickly noticed that my code felt much more elegant: changes to data were entirely separated from changes to different views on that data, e.g. changing user data used by both the messages view and a user list view, or changing message data used both by a message replying to it and the message itself. The models themselves also were reduced in complexity: messages model only has to worry about ordering and adding/removing IDs from the list as Telegram dictates, and the messages store only has to worry about adding and notifiying about updates from Telegram. No more “when message changed, try to locate the child model for the chat that that message belongs to and post the event there if it's present, otherwise don't.” It's simply “when message received, change the message store.”

Conclusion

After using relational data, I strongly agree with my friend's bewildered reaction to me saying there was nothing like relational state in Qt. Many things that used to be “how do” or mistake-prone boilerplate hells for me now have readily apparent solutions for me now.

Contact Me

If you have any thoughts on this post, feel free to share them w/ me in #chat:kde.org or https://t.me/kdechat, or by DMing me at @pontaoski on Telegram or @pontaoski:kde.org on Matrix.

There's a pair of antipatterns I've ran into when dealing with QML (QtQuick & Qbs) that I keep seeing people make a lot; so I might as well make a blog post detailing what they are and why they're bad.

property alias

property alias is a pretty surefire way of getting further from “single source of truth” which translates into spaghetti QML where you have to dive through things chained to each other like plumbing to an amount that negatively affects readability.

Instead of:

Item {
	property alias text: label.text
	Label {
		id: label
	}
}

you should:

Item {
	id: root
	property string text

	Label {
		text: root.text
	}
}

Not only does this centralise all the state for an item and make it easier to locate, it also has the effect of encoding the property's type with the property declaration itself, making it easier for tooling (and humans!) to identify the type of the property. This is particularly important for public-facing APIs, where API documentation tools cannot evaluate the QML to determine the underlying type of the alias.

id overuse

id is extremely easy to abuse. In short, id has a very limited possibility space before it starts to become spaghetti: reading user input status from controls and referring to a view's root.

This mostly has to do with state management in general: “single source of truth” is the key; avoiding fragmenting your program's state throughout random items in a view mostly leads to more readable QML, and provides other benefits as well. For example, a chat application whose chat view has a single source of truth to the current navigational state can easily save/load where it was. If you're passing a full ChatModel* object to your messages view when you push() instead of an ID that allows the messages list view to obtain a ChatModel* itself; that isn't SSOT, as the ChatModel* has to come from somewhere in the state of the chat list view pushing the new messages list view, and it means that other views cannot push a new messages view view unless they replicate the code & state that the chat list view had; which is prone to bugs.

What does id have to do with this? id allows fragmenting your single source of truth by making it convenient to put stateful properties in scattered objects and refer to them from anywhere in a context/scope. Locating what does what when many child contexts are able to mutate a random object in a non-centralised manner is painful, to say the least.

Some QML frameworks provide facilities for making “single source of truth” and reducing id usage easier to perform; e.g. the Kirigami.PageRouter which can facilitate SSOT with the route API allowing centralising data into a singleton that stores navigational state, which applications should use as the only information needed to fetch other things like network resources. In Qbs land, file tags + the Project type serve this purpose; the Project type centralises various Products into a single source of truth for managing them, and file tags are the source of truth for input/output and composing of various Rules.

So, if you're reading this, you probably need some background information. gRPC is a popular RPC system based on HTTP2 and Protobuf. hRPC is an RPC system we at Harmony are porting to from gRPC, built on HTTP1 and Protobuf as well. Harmony is a chat protocol that falls somewhere in between Matrix and Telegram functionalitywise with a ton of extra goodies (besides our in-progress E2EE draft, which is basically just a micro Matrix statewise implemented with Protobuf instead of JSON.)

(If you're reading this on the KDE planet, there's juicy Qt stuff well after we explain what the heck all this networking stuff is, don't worry :) )

gRPC: The Good

gRPC has substantial language support, and is widely available in distros. It's also extremely optimised, using substantial custom HTTP2 behaviour for minimum network transfer.

gRPC: The Bad

gRPC has a very, very big flaw for publically facing services: streams play awfully with reverse proxies like nginx, as they're essentially HTTP2 requests that aren't closed. This causes proxies to be like “hmmm this is a slow loris attack, time to yeet this stream.” For our homeserver at https://harmonyapp.io, this means we had to configure nginx to be ok with requests taking an entire hour. Any streams would always terminate at exactly 60 minutes. To be fair to gRPC, there's a dedicated HTTP2 streams thing being worked on that would allow reverse proxies like nginx to play nice with it, but unfortunately that's not the case now.

Besides that, gRPC's client libraries, while widely available, range from mediocre to [ censored ] awful. gRPC is a Google product that isn't Go, which means that “error handling” is not a word in its dictionary. This has really bad implications for the C++/Qt client, Challah. Essentially, if anything goes marginally wrong, the client just straight up aborts. There is no way for us to gracefully recover from any errors that originate from the gRPC library. This is terrible for the user experience, as we can't even show a “something is going wrong” page. This is one of the big reasons we're moving away from gRPC: we cannot have our only desktop client be crashing on anything slightly less than perfect network conditions.

That wouldn't be a problem, if making our own implementation of gRPC was easy. Unfortunately, it's not. Remember the part where I said it used low-level HTTP2 a lot? Yeah, that gets very complicated very fast.

Additionally, our web client, Tempest cannot do the said low-level HTTP2 stuff. This requires us to specify in the protocol documentation a place for servers to name a grpc-web proxy for web clients to use.

With all of these issues (for our usecase specifically; none of these would affect its usage with microservices which seem to be the main reason people use gRPC) in mind, we knew that using gRPC wouldn't cut it if we wanted something as polished as we hoped. And thus, we started hRPC.

The Goals

We decided quickly that hRPC should: – require minimal if no changes to our .proto files – be dead-simple to implement – be web-compatible (which basically means HTTP1/WebSockets)

The Implementation

First things first, we needed to write a protoc plugin. Thankfully, that was simple. We decided to use a hybrid approach: simple to generate languages like Go would be done using Go's text/template package to write templates, which could either be packed into the binary or loaded from external files on disks to facilitate third parties writing their own templates. Complex to generate languages like Qt/C++ would be done using dedicated functions in protoc-gen-hrpc. This was actually so simple that we decided to write another plugin, protoc-gen-hdocs which generates our online reference documentation from the .proto files. Our JS client doesn't need to make use of this; as the protobuf implementation in JavaScript is transport agnostic. Blusk, the other lead developer of our project, simply wrote a function that takes request information + the inputs and transforms it into the outputs. Likewise, our Rust client and SDK makes use of its own code generation instead of the protoc plugin. That leaves our Go server/client and our C++ client as being generated by protoc-gen-hrpc.

The Flaws

Not everything is rosy with hRPC. Due to using HTTP1 and the straightforward solution to networking, our implementation is nowhere near as slim on networking as gRPC which spends a lot of time and effort shedding bytes. Additionally, we're forgoing the existing gRPC ecosystem, requiring anyone that wants to implement the Harmony protocol to write their own codegen that works with our .protos.

Challah: gRPC

Challah, our Qt/C++ client, uses QtConcurrent approximately a heck tonne to handle sending gRPC requests and receiving data from streams without blocking the main thread. In short, every request uses a thread from a thread pool to send, and depending on its nature, either uses a callback, or will result in data coming into the “main” events stream, in which case we don't take a callback. For streams, we use two tools:

• thread doing a busy loop constantly doing a while (stream->Read())
• events

Events are amazing, and allow us to use a largely mutex-free design. What happens is that our stream-reading thread will read the events stream, and translate gRPC reads into Qt events, which are posted to parents of our object hierarchy, which then re-post the events to their children as necessary. The flow for a message looks like “client thread reads event, posts event to communities model, which posts event to its child channels model, which posts event to its child messages model, which then updates data.” Some of this is working around gRPC, but is mostly a sane concurrency and state management solution on its own, which will mostly be intact with the port to hRPC.

Challah: hRPC

Now that I'm the one authoring the client RPC library, I get to make it perfect as possible for Challah. That means it's written with Qt and uses its proper concurrency mechanisms. Fun stuff :). Besides shedding a runtime dependency, the port to Qt will also massively help with portability: gRPC C++ is a giant and clunky beast with a lot of vendored dependencies. Currently, our macOS build isn't working (though it compiles) due to SSL woes. Qt's networking stuff lacks those woes, and moving to them will mean that our macOS build will begin functioning. This also opens the room to an Android build, as protobuf library will be substantially easier to pack for Android than gRPC. Codewise, this means that we can port from abusing QtConcurrent thread pools to just using Qt's native networking types. All in all, that's pretty good.

I'm not sure how to end this blog post, so I'll just drop some links:

• hRPC (codegen + docs gen repo): https://github.com/harmony-development/hrpc
• Challah (Qt/C++ client): https://github.com/harmony-development/challah
• Crust (Rust client): https://github.com/harmony-development/Crust/tree/feat/hrpc
• Rust SDK: https://github.com/harmony-development/harmony_rust_sdk/tree/feat/hrpc

Maybe I'll write some more about how implementation of the codegen or the C++ client worked if I see that this post is somewhat popular or if someone requests it. Tschö.

Tags: #libre

The base class of your item affects a large amount of your public-facing API, since everything that base class offers will also become part of your public-facing API, and you'll need to ensure that

1) the public-facing API remains cohesive 2) your component is a logical subclass of the base class

Cohesion

What exactly is API cohesion, anyhow? Cohesion, as per the dictionary is “the action or fact of forming a united whole.” A cohesive API has two main requirements

1) all of the parts make up the complete functionality of your API 2) all of the parts make up only the complete functionality of your API 3) none of the parts conflict

Complete Functionality

When designing an API, you should first and foremost outline what the thing does. Make sure to account for all of the possible usecases you can think of ahead of time, and you can always revise your API to be more ergonomic or featureful after you release it. A cohesive API should preferably achieve as much as possible with as little as possible, even though a large API can be cohesive. Too big of an API and developers will have trouble getting acclimated to it. If your API fails to offer functionality that the user would expect from it, it loses cohesiveness as the developer has to cobble together something on their end in order to implement the functionality the developer expected.

Only The Complete Functionality

When implementing an API, you need to keep in mind that you should only expose what the API of that component should offer logically. If you inherit from a base class that exposes too large of an API surface, your component may have stuff that doesn't make sense. For example, your component may be clickable, so you decide to make it a subset of a Button in order to gain clickability-related things. This, however, is likely to introduce stuff into your API that doesn't make sense. For example, a control inheriting from Button and lacking room in its design for an icon, e.g. a clickable image, would have to deal with the icon grouped property of Buttons. Since the icon grouped property does not form part of the implemented API, yet is part of the API the end user gets, this makes the API not cohesive. Such a control would also have issues with the text property of Buttons. This combined with other issues makes inheriting from a Button a poor choice for a “clickable image” component's API despite both buttons and clickable images being clickable UI components, since it causes the API design to not be cohesive.

No Conflicts

Blindly inheriting from a base class can cause two parts of the API to conflict: one that you provide in your subclass, and one coming from the superclass. For example, say you're making a Button-based control that can take multiple actions, e.g. a floating action button that reveals more buttons corresponding to actions when tapped. Whilst implementing this API, you add a primary action and a list of secondary actions. This introduces an API conflict due to not considering the base class's own offering—the action-based API of the expanding FAB conflicts with the button's own API: one action and some signals to connect to. This is one of the worse cases of an incohesive API—due to there being many ways to perform the same action, developers will likely use something outside of what you designed for, and will run into places you didn't implement. Even if you accomodate for these conflicts, e.g. handling both action and primary in the FAB component, you'll still have multiple ways to achieve the same goal, which is still not a cohesive API, and makes it harder for multiple developers to come to a consensus on how to achieve something with your API—not good.

Logical Subclasses

Determining whether something or not is a logical base class of your component should be accounted for when implementing your API. For example, you may want to render something with a foreground and a background internally, so you opt to make your component a Control, since it has a foreground and a background property, and automatically lays them out in a desirable manner. However, your component may not necessarily make sense as a Control APIwise, even if you want some of the functionality to implement it—say you're just using the background and foreground for visual effect only, as in a UI element that renders an user avatar. A component like this has no use for most of the API added by a control—you're using the component for its visual look, not its functionality, so overriding the visuals whilst keeping functionality intact with a public background/foreground property doesn't make sense. Fonts, spacings, UI colour palettes, and other aspects of a Control do not make sense for this component, therefore the base class should not be a Control, but rather an Item containing a Control internally. When in doubt, an Item with implicitHeight/Width set and manually exposed properties is often a better choice APIwise than exposing all of the features of the class you're using to implement the API, but not the class whose functionality you want as part of the public API. In short, pick parent classes for the API you want to expose, not the implementation you want to make. Do not expose implementation details.

sina wile kama sona e poki ilo anu seme? pona mute a!

nanpa wan: wile sona

sina wile e poki ilo la sina wile e ilo. sina wile e poki e seme? o kama sona e ona.

lipu sona ni li pana sona e ni: poki e ilo Toki Gnu.

nanpa tu: ilo

sona wile poki e ilo la sina wile e ni:

dnf install rpm-build rpmdevtools

kulupu ilo Rpm ale li kepeken e ilo ni.

o ni:

rpmdev-setuptree

nanpa tu wan: lipu Rpm

jan li ken poki e iko kepeken lipu Rpm.

ona li lukin e ni:

nimi: ijo

pali %{ilo lili}

nanpa tu tu: kama ilo

o ni:

cd ~/rpmbuild/SOURCES
wget http://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz

tenpo kama la lipu mute li kama.

Tags: #libretpo

Yes, that title is too long and I know it.

If my previous blog post didn't make it clear, I don't like dealing with XML. Obtuse to write, obtuse to read. Given that I wrote a program so that I wouldn't need to write XML for an application menu protocol, it only makes sense that I would do the same for reading Wayland protocols. And thus, ReadWay and its non-web cousin ilo Welenko were born.

Parsing the XML

If you're familiar with Wayland, you're probably familiar with the XML files you can find in /usr/share/wayland and /usr/share/wayland-protocols. What you may not have noticed is the /usr/share/wayland/wayland.dtd file lurking alongside the core Wayland protocol. This is a document type definition file, which defines what a valid XML document looks like. Thankfully, this is a fairly simple DTD to write Go structures for. This DTD definition:

<!ELEMENT description (#PCDATA)>
  <!ATTLIST description summary CDATA #REQUIRED>

becomes this Go code:

type Description struct {
    Summary string `xml:"summary,attr"`
    Body    string `xml:",chardata"`
}

And this:

<!ELEMENT protocol (copyright?, description?, interface+)>
  <!ATTLIST protocol name CDATA #REQUIRED>

becomes

type Protocol struct {
    Name        string      `xml:"name,attr"`
    Copyright   string      `xml:"copyright"`
    Description Description `xml:"description"`
    Interfaces  []Interface `xml:"interface"`
}

Fairly simple, eh?

To unmarshal a protocol XML into a Go structure, you just xml.Unmarshal like this:

data, err := ioutil.ReadFile(path)
// handle error
proto := Protocol{}
err = xml.Unmarshal(data, &proto)
// handle error
// do something with proto

Templates

Of course, Go structs aren't particularly easy to read for documents even compared to XML. This is when Go's html/template package comes into play. You can throw a Protocol and a template at it like so:

<h1>{{ .Name }} <small class="text-muted">protocol</small></h1>

<p>
    {{ .Description.Body }}
</p>

{{ range $iface := .Interfaces }}
    <h2>{{ $iface.Name }} <small class="text-muted">interface version {{ $iface.Version }}</small></h2>

    <!-- finish rendering interfaces -->

{{ end }}

Of course, you have the more generic text/template package, which is what ilo Welenko uses. Same concept applies:

Kirigami.Page {
    title: "{{ .Name }}"
    ColumnLayout {
        {{ range $iface := .Interfaces }}
        Kirigami.Heading {
            text: "{{ $iface.Name }} version {{ $iface.Version }}"
        }
        {{ end }}
    }
}

(And yes, I am statically generating QML code in Go and loading it instead of marshalling it into Qt data types and using model/views/repeaters.)

See Also:

• ReadWay hosted: ReadWay hosted on the internet. The “special thing that might happen when you drag an XML file onto [the] paragraph” is a Wayland protocol being rendered in your browser using WASM. The future is now. And it don't need no cookies.
• ReadWay source: The static generator for ReadWay.
• ilo Welenko: The desktop counterpart to ReadWay that renders into QML rather than HTML. At the time of this post, it's very incomplete compared to the web version.

Contact Me

Have any thoughts/comments/concerns about this post, or want to tell me that I shouldn't statically render QML? Here's how you can contact me:

• Telegram: @pontaoski
• Discord: pontaoski blackquill 🏳🌈#8758
• Matrix: pontaoski@tchnics.de
• IRC: appadeia_
• Email: uhhadd@gmail.com

Tags: #libre