Concurrent Data Processing In Elixir (B4)

Now that the book has reached the sent to print stage I am again working through it.

Here is the repo containing the samples from the first chapter:

The first chapter works through the simplest solution to concurrency: Tasks.
It starts with a simple slow example and step by step adds more sophistication until you get to have a Task.Supervisor watching the code.

This is one of the more elegant examples of demoing the power and simplicity of Elixir tasks that I have seen.

Some Useful Aliases

I have started to add custom aliases to my bash setup to reduce typing time:

alias ism="iex -S mix"

This is great for turning code into an interactive session

alias mtp="mix test --include pending"

This allows quick testing of Elixir Exercism exercises.

Testing Your System As A Chain

A given software system is a series of components that need each other to work. To test it you can either test all of it or you can treat it like a chain and test each link.

The current system I work on has a database, a Java backend, JSP views and uses AngularJS in the front-end

In order to comprehensivly test it you need a range of components. Featupent re flags can be tested by asserting that the page has certain elements present or absent. You can test for the presence of a given angular tag.

The angular components can be tested in a unit test framework in isolation.

Protractor is useful for checking that the components hang together without errors. There are so many places that JavaScript can live that only by looking for errors on the logs or ensuring that the page still has interactivity can you be sure you have s functioning page.

I spent some of last year looking for these assertiobs in React only to find them in an angular library.

Using Livebook as a Model

I have been having a quick experiment with livebook.

This is an interactive website that can be used to easily share Elixir code.
Think of it as Juypter Notebooks but with the backing system being Elixir rather than Python.

It has a very neat docker image:

docker run -p 8080:8080 livebook/livebook

This will stand up a local website running livebook. The console will give you a link (including a token) to allow you to access it.

You can create notebooks and add sections in Markdown or Elixir.
The great thing is that these can be interactively edited by multiple concurrent users.
You can also export a notebook as a file, publish the details and import it into another livebook.

I have been thinking about using this to recreate an old mac simulation tool. This tool allowed you to build a model with data flowing through it and you could use it to see bottlenecks.

I have three code blocks:

The first is the initialisation block

Agent.start_link(fn -> 0 end, name: :complete)
Agent.start_link(fn -> 0 end, name: :iteration)
Agent.start_link(fn -> 0 end, name: :qa_backlog)

The second is the update block

Agent.update(:iteration, fn x -> x + 1 end)
Agent.update(:complete, fn x -> x + min(10, Agent.get(:qa_backlog, & &1)) end)
Agent.update(:qa_backlog, fn value -> max(value - 10, 0) end )
Agent.update(:qa_backlog, fn value -> value + 20 end)

The third is the output block

:ok = IO.puts "Iteration #{Agent.get(:iteration, & &1)} Backlog #{Agent.get(:qa_backlog, & &1)} Complete #{Agent.get(:complete, & &1)}"

If you put the three blocks into livebook then you have the start of a model.

This is a very simplistic model. It attempts to show that the QA team will never be able to keep up with the two dev teams feeding it work. This could become far more sophisticated (with a random percentage of QA items failing and being returned to the teams). The idea here is to show how you can use an Agent in livebook.

I am going to see if you can add visualisations.

Programming Phoenix LiveView – Part 1

I have now started work on the Phoenix LiveView Book.

Still early days on this book.

However this will make creating a phoenix app much easier:
docker run -d -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:11

That makes a default password postgres image.
I would not recommend default passwords for anything other than local demo databases.
This makes setup much less painful.

Here is the repo that I am working on:

Patching AngularJS

2021 is a very unusual time to be returning to use AngularJS. This is the version of Angular that is going out of support at the end of this year! However I have a client that has a very large codebase that needs to be moved to the latest stable version before they start the move to a more modern Angular version.

A key feature of AngularJS is that it is built using a dependency injection framework. This means that it is possible to replace parts of the supplied libraries with alternative versions. For this codebase this can mean an older version of one of the system components. For example the $http component changed it’s call interface between 1.5.6 and 1.6.0. It would be great to move the entire codebase to the more standards compliant promise library, but this is just not feasible for the size of the codebase. There are thousands of uses of the old syntax.

This does allow us to take advantage of working on a library that has been abandoned – we know all of the changes that the later versions have made and also know what the replacement looks like. This gives us a chance to look at the direction we need to take the code before we migrate. This is not going to be quick.

Stop Fighting The Last War

There is a classic military problem at the start of a new campaign. The goals and missions initially attempted are based upon the doctorine that worked in the previous campaign. This can have disastrous effects. During the first Gulf War this resulted in a convoy of tanks left inoperable at the side of the road. A key filter needed to be replaced to allow use in a desert.

I am shortly to start on a new project at a new client in a very different domain to any that I have worked in before. This will involve checking all of the assumptions that I am about to make.

The previous engagement had poorly defined goals. This is one point that can be corrected early on for the next project. What is the problem we are trying to solve? How do we measure progress towards this goal? What are the constraints here?

Rate Limiting Lambdas

Recently I managed to accidentally exhaust the pool of concurrent lambda executions on an account.

We have an S3 bucket full of files that we need to parse and store in a database. They get parsed by a trigger when added. We have added some code that can extract extra details from the files. The naive approach is to have a scheduler trigger the Lambdas at a fixed rate. The problem was that due to another problem the database became overloaded. This led to a sorcerer’s apprentice style of retry overload.

We now use reserved concurrency combined with an SQS queue to limit how many messages are processed at a time.

Reserved concurrency takes some of the concurrent lambda capacity (which defaults to 1000 per account) and restricts a given lambda to use that. I would prefer the ability to have a cap rather than dedicated capacity, but I can see why it is there.

I have taken advantage of the long weekend to tune the values of timeout and reserved capacity so that I don’t overload my Postgres database. Thread pooling is hard when you have Lambdas.

Lesson learned:

– Lambdas are not an unlimited resource.

– Don’t have too many unrelated services in the same account

– SQS plus Lambda with a dispatching lambda allows effective rate limiting.

Software Design X-Rays

I have started studying this book.

To make life easier here is the link to the examples:

So far I agree with the analysis. One minor niggle is the claim that small config files don’t need tests. That’s not true for say package.json.

My dependabot usage results in dependabot making 60 times as many commits as the next user. You really need a unit test for every feature that you use in a dependency. In the node world that can be painful.

First Looks at Tailwind CSS

I am working through the book Modern CSS with Tailwind.

Tailwind is a css framework for styling a website.
It is strongly opinionated on certain things.
To start with it includes a complete reset library that removes all browser default styling for all HTML elements.
The markup is effect describing rather than semantic:

text-gray-300 hover:text-gray-700

This is intended to make the markup very easy to understand, at the price of being verbose.

Tailwind CSS 2.0 has taken the decision not to support Internet Explorer 11. This will allow a lot of modern (or technically by now non-ancient) CSS techniques to be used. This would prevent me from using it on my current client who still has a 1/6 of the users reporting as ie 11.

Microsoft have made some announcements about their support for IE11:

Microsoft teams has already stopped supporting IE11 as of November 2020.
Microsoft 365 (the current name for the online office suite) will drop IE11 support in August 2021.
It will be interesting to see how quickly the big corporates drop IE11 support.