Using Livebook as a Model

I have been having a quick experiment with livebook.

This is an interactive website that can be used to easily share Elixir code.
Think of it as Juypter Notebooks but with the backing system being Elixir rather than Python.

It has a very neat docker image:

docker run -p 8080:8080 livebook/livebook

This will stand up a local website running livebook. The console will give you a link (including a token) to allow you to access it.

You can create notebooks and add sections in Markdown or Elixir.
The great thing is that these can be interactively edited by multiple concurrent users.
You can also export a notebook as a file, publish the details and import it into another livebook.

I have been thinking about using this to recreate an old mac simulation tool. This tool allowed you to build a model with data flowing through it and you could use it to see bottlenecks.

I have three code blocks:

The first is the initialisation block

Agent.start_link(fn -> 0 end, name: :complete)
Agent.start_link(fn -> 0 end, name: :iteration)
Agent.start_link(fn -> 0 end, name: :qa_backlog)

The second is the update block

Agent.update(:iteration, fn x -> x + 1 end)
Agent.update(:complete, fn x -> x + min(10, Agent.get(:qa_backlog, & &1)) end)
Agent.update(:qa_backlog, fn value -> max(value - 10, 0) end )
Agent.update(:qa_backlog, fn value -> value + 20 end)

The third is the output block

:ok = IO.puts "Iteration #{Agent.get(:iteration, & &1)} Backlog #{Agent.get(:qa_backlog, & &1)} Complete #{Agent.get(:complete, & &1)}"

If you put the three blocks into livebook then you have the start of a model.

This is a very simplistic model. It attempts to show that the QA team will never be able to keep up with the two dev teams feeding it work. This could become far more sophisticated (with a random percentage of QA items failing and being returned to the teams). The idea here is to show how you can use an Agent in livebook.

I am going to see if you can add visualisations.

Programming Phoenix LiveView – Part 1

I have now started work on the Phoenix LiveView Book.

Still early days on this book.

However this will make creating a phoenix app much easier:
docker run -d -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:11

That makes a default password postgres image.
I would not recommend default passwords for anything other than local demo databases.
This makes setup much less painful.

Here is the repo that I am working on:

Patching AngularJS

2021 is a very unusual time to be returning to use AngularJS. This is the version of Angular that is going out of support at the end of this year! However I have a client that has a very large codebase that needs to be moved to the latest stable version before they start the move to a more modern Angular version.

A key feature of AngularJS is that it is built using a dependency injection framework. This means that it is possible to replace parts of the supplied libraries with alternative versions. For this codebase this can mean an older version of one of the system components. For example the $http component changed it’s call interface between 1.5.6 and 1.6.0. It would be great to move the entire codebase to the more standards compliant promise library, but this is just not feasible for the size of the codebase. There are thousands of uses of the old syntax.

This does allow us to take advantage of working on a library that has been abandoned – we know all of the changes that the later versions have made and also know what the replacement looks like. This gives us a chance to look at the direction we need to take the code before we migrate. This is not going to be quick.

Stop Fighting The Last War

There is a classic military problem at the start of a new campaign. The goals and missions initially attempted are based upon the doctorine that worked in the previous campaign. This can have disastrous effects. During the first Gulf War this resulted in a convoy of tanks left inoperable at the side of the road. A key filter needed to be replaced to allow use in a desert.

I am shortly to start on a new project at a new client in a very different domain to any that I have worked in before. This will involve checking all of the assumptions that I am about to make.

The previous engagement had poorly defined goals. This is one point that can be corrected early on for the next project. What is the problem we are trying to solve? How do we measure progress towards this goal? What are the constraints here?

Rate Limiting Lambdas

Recently I managed to accidentally exhaust the pool of concurrent lambda executions on an account.

We have an S3 bucket full of files that we need to parse and store in a database. They get parsed by a trigger when added. We have added some code that can extract extra details from the files. The naive approach is to have a scheduler trigger the Lambdas at a fixed rate. The problem was that due to another problem the database became overloaded. This led to a sorcerer’s apprentice style of retry overload.

We now use reserved concurrency combined with an SQS queue to limit how many messages are processed at a time.

Reserved concurrency takes some of the concurrent lambda capacity (which defaults to 1000 per account) and restricts a given lambda to use that. I would prefer the ability to have a cap rather than dedicated capacity, but I can see why it is there.

I have taken advantage of the long weekend to tune the values of timeout and reserved capacity so that I don’t overload my Postgres database. Thread pooling is hard when you have Lambdas.

Lesson learned:

– Lambdas are not an unlimited resource.

– Don’t have too many unrelated services in the same account

– SQS plus Lambda with a dispatching lambda allows effective rate limiting.

Software Design X-Rays

I have started studying this book.

To make life easier here is the link to the examples:

So far I agree with the analysis. One minor niggle is the claim that small config files don’t need tests. That’s not true for say package.json.

My dependabot usage results in dependabot making 60 times as many commits as the next user. You really need a unit test for every feature that you use in a dependency. In the node world that can be painful.

First Looks at Tailwind CSS

I am working through the book Modern CSS with Tailwind.

Tailwind is a css framework for styling a website.
It is strongly opinionated on certain things.
To start with it includes a complete reset library that removes all browser default styling for all HTML elements.
The markup is effect describing rather than semantic:

text-gray-300 hover:text-gray-700

This is intended to make the markup very easy to understand, at the price of being verbose.

Tailwind CSS 2.0 has taken the decision not to support Internet Explorer 11. This will allow a lot of modern (or technically by now non-ancient) CSS techniques to be used. This would prevent me from using it on my current client who still has a 1/6 of the users reporting as ie 11.

Microsoft have made some announcements about their support for IE11:

Microsoft teams has already stopped supporting IE11 as of November 2020.
Microsoft 365 (the current name for the online office suite) will drop IE11 support in August 2021.
It will be interesting to see how quickly the big corporates drop IE11 support.

Concurrent Data Processing In Elixir (B2)

This book works you through the various concurrent programming techniques available to Elixir.

This is still in Beta, so not yet finished (and has the last few typos to be fixed), but is a great introduction.
It even covers some of the recently added features of the venerable GenServer so you get to see how to use handle_continue which is great technique that allows a genserver to return and still carry on working on a problem.

It starts with Tasks, introduces GenServers.
Then it moves onto GenStage (a system for using Backpressure to control the flow of data through an application.
Flow adds wrappers around GenStage to simplify the usage (it’s almost as easy as Streams).

The examples are mostly contained within individual chapters which makes picking up a topic much easier than other PragProg books that I have worked through.

Awaiting the B3 release!

Heisenberg And Death March Projects

A simple explanation of the Heisenberg Uncertainty Principle goes as follows:

You have a particle that you are trying to find out where it is an how fast it is moving. You are taking measurements by shining a laser onto it. If you want more accuracy on the location then you can shine a brighter laser. However this changes the velocity. If you want a more precise velocity then you need to use a dimmer laser. There is a finite limit to the combined accuracy or position and velocity.

There is a similar problem with projects that are deemed late. Frequently the solution is to add more frequent and intensive status reports. The problem is the addition of the status report takes away time that could be spent working so it will actually delay the result. In extreme cases the project will then never be finished as all time is taken up by reporting.

Learning Go

Later this week I am going to be assessing a developer on a Pair Programming TDD Programing Exercise in Go.
The problem is that I currently don’t know Go.

I do own the book Introducing Go from a Humble Bundle.

These are the notes that I have from working through the first few chapters.
I am basing the analysis upon the 7 Languages in 7 Weeks pattern.

Go is strongly typed.

Bracket style is K + R (opening bracket stays with the function declaration)

Integer size is machine (platform dependent)
Method names seem to use PascalCase.
Variable names use camelCase.

Variables can only be defined once per scope.
Variables are mutable.

Declared by var
The type follows the name if required:

var x string

I have not seen this in recent languages (this is a Pascal idea rather that the C style of type name)
Types can be implied by assigning with := (but only inside a function).

const allows the creation of things that cannot be reassigned.

Backticks allow for multiline strings.

Compiler warns about unused variables.

Variable names can be redefined at a different scope.

No implicit returns from functions. (Which is unusual for a modern language).
You can name the return variable, assign it and then call return.

You can return tuples from a function.

Does not handle overflows well.

func factorial(x uint) uint {  
    if x == 0 {  
       return 1  
    return x * factorial(x-1)

This fails on 100, and will return 0

It appears that a function cannot be overloaded.

Interfaces are implicit. You don’t need to state that you are using it.

The early versions of Go were really weak at package management. It requires you to have your code in a sub folder of the GOPATH called src. This completely insane! The language should not define the name of the folder structure.

Tests need to be named Test* and have a parameter of t *testing.T

I think I now know enough for the Pairing Interview.