Schema based Development: Protobuf and Avro

~ chriseyre2000 ~ Leave a comment

Back in the year 2000 there was a concept around that could have changed development. WSDL schemas allowed a project to define the contract that a service provided. Admittedly most people reverse engineered this from their existing systems and WSDL generated huge SOAP monstrosities. Having a single source that described a contract for a system really helps with integration. Making the contract a defined versioned item ensures that accidental breakages are less frequent.

In other spaces there were similar solutions that got more useful traction. Google had the problem of a large number of systems that needed to interact. They also used a range of programming languages to do this. Their solution was Protobuf. This is a file format that defines the serialization format for messages. Each service would use Protobuf to define the contract and libraries would generate the code to serialize/deserialize from the wire-protocol. It has strict versioning rules:

  • Identifiers cannot be reused (that is the underlying numeric identifier)
  • You can add a field to an existing read contract without breaking anything.
  • A required field may become optional or repeating, but the reverse is not allowed.
  • If you need more changes than the above allows you increment the version of the contract.

Oddly this is also the rules enforced by google BigQuery. Either it is isomorphic to Protobuf or uses Protobuf under the hood!

There are several styles of protobuf clients, some of which are smart enough to allow the generation of deltas. This means that when a large message changes and the client already has the previous version you can only send the updates. This proved to be very useful.

A typical usage of Protobuf is for the client to conform to the contract supplied. However it is a great system boundary and it is possible to use this at the edge of Bounded Contexts to translate field names into local concepts.You can also chose to ignore data that you don’t want.

Avro is a similar tool, typically used in the Kafka ecosystem. Here it is common to keep the schema in a Schema registry. This makes the asymmetric approach more difficult.

Craft GraphQL APIs in Elixir with Absinthe Part One

~ chriseyre2000 ~ Leave a comment

I have finally started working through this book. The downside of using an older book is that the samples use what is now ancient versions of Phoenix. I could install the older tools and try and make them work. This may or may not work due to distributed code rot (reliance on a library that no longer exists). Instead I have started from scratch.

Here is the repo that I am working on: https://github.com/chriseyre2000/absinthe_demo

My setup is simple.

I am using docker for postgres:

docker run -d -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:11

Don’t do this in production, but it’s a quick way to stand up a demo.

With this I created a simple phoenix application:

mix phx.new a1_new

I have added the following to mix.exs:

      {:absinthe, "~> 1.7.0"},
      {:absinthe_plug, "~> 1.5.8"},
      {:absinthe_phoenix, "~> 2.0.2"},
      {:absinthe_relay, "~> 1.5.2"},

I copied over the repo migration scripts

mix ecto.create

mix ecto.migrate

I have also copied the default Schema to the new project.

So far this is enough to get the lookup_type working, once I noticed that you need to query form MenuItem, not menu_item (a level 8 network error).

I also needed to copy over the Ecto Schema from the sample (and fix up the module name).
Note that null: false is no longer valid Ecto syntax (in fact never has been, it was just not raised as an error before).

The Typescript Trap

~ chriseyre2000 ~ Leave a comment

Typescript is a distinct improvement over JavaScript in that you can find a whole class of problems early. This can lead to the assumption that you need less testing, which is unwarranted.

Typescript is strictly a preprocessor. The types only exist at compile time. This can result in a system that is capable of getting into states that the compiler would imply are impossible.

I have recently been working with GraphQL and the Relay compiler. In one example I had a type with a field marked as not null. Due to some interesting decisions of the Relay caching (https://github.com/facebook/relay/issues/2237 performance is preferred over accuracy) it could set the non-null field to be null. This was due to some aggressive caching where a query in an unrelated component could replace the value returned. This causes non-local bugs, requiring extensive analysis and testing. One of the possible solutions to this is for the Relay compiler to remove all non-null contraints – something that may be accurate but will require excessive null checks!

Typescript is not magic, you do need to test the application and at an aggregated level.

Has Boris Johnson Actually Resigned Yet?

~ chriseyre2000 ~ Leave a comment

Yesterday Boris Johnson gave a speech which has been claimed to be his resignation. At no point did he state that he had resigned either as the leader of the Conservative Party or as Prime Minister.

He did state that a leadership election for the Conservative Party would shortly be underway and that he would hand over to the new leader in the Autumn. Given that we now know that resigning ministers get three months pay it seems as if Boris is going to drag out the process so that he will be paid until the end of the year.

There are a number of investigations into his activities that need to take place (largely to do with his alleged improper Russian interactions). He cannot remain in office while they are happening. In any other setting he would be placed on suspension or put on gardening leave.

Boris Johnsons premiership has demonstrated holes in the British Constitution. Ministers are expected to act honourably and resign if their honour is called into question. This seems to have fallen out of practice. Given the weight that is given to president this cannot be left to stand.

Progressive Failure Modes

~ chriseyre2000 ~ Leave a comment

Any part of the system you are working on has a primary focus. It may also have secondary features.

In simple terms if you are selling books searching for a title and buying a specific book are the primary focus. Secondary features include sales counts, lists of reviews, you also may like details.

A component of a system has dependencies on sources of data. These are not always reliable. If you can still fullfil the primary focus it should be possible to present a stripped down version when the secondary features fail. Only if the primary feature of the component is failing should it report a direct error to the user.

This is similar to the concept of a mobile first progressive web app where basic features work in simple devices and additional options are layered on as screen size increases.

Building systems with this in mind will result in improved stability. Fancy add-ons should not compromise the basic functionality.

Thoughts on Rules Engines

~ chriseyre2000 ~ Leave a comment

Some years ago I built a rules engine system for my employer. This is the theoretical underpinning.

There are various types of rules that a rule engine can have. The most basic is the default. This will apply a value to an initialised but not updated field.

The next is a calculation. This will set a value typically based upon data in one or more locations. This will recalculate whenever either value changes.

Another is the non-limiting constraint. This will warn that a value is somehow incorrect but will not change values.

This contrasts with the limiting constraint that will prevent a value from violating it’s rules, enforcing a limit should the constraint be broken.

Relay compiler and VSCode

~ chriseyre2000 ~ Leave a comment

If you are using relay to make your graphql development easier (it generates typescript types from your graphql schemas). This can make working with VSCode difficult as the IDE sometimes has a different view of the world to the latest generated version.


I have found the following extension for VSCode: Relay for VSCode I plan to try this tomorrow to see if it helps.

This may also help testing the components that use relay: https://relay.dev/docs/guides/testing-relay-components/

It’s impossible to secure the browser!

~ chriseyre2000 ~ Leave a comment

If you are exposing an endpoint to your front end on the public internet then it can be called at any time. There is no way to ensure that the call came from a browser. The only sane approach is to assume that any endpoint that you are exposing to the browser is a public API and should be treated as such (with the exception of ensuring backwards compatibility – private API’s are still subject to change/retirement without notice).

This also means that any javascript that you embed in the page and download to the browser will include a map of your exposed backend services and examples of how to authenticate and use it. Obfuscation will only go so far – the urls struct that you send out will be included in the messages.