Recently I have been turning to Elixir to solve some personal projects. It is a great general purpose language.

Like a lot of people I am mostly working from home. This involves working over a VPN to access some of the essential services that the client uses. The VPN client infrequently disconnects which is typically only noticed when one of these services become inaccessible.

It would be useful to be notified that this has happened. I found another Elixir project (https://github.com/navinpeiris/ex_unit_notifier) that had solved the notification problem but specifically for ex_unit. The trick is to install a CLI notification tool and execute that with appropriate parameters.

Once I had this I could use a simple webclient (HTTPoison) to check that a typical page exists that is only available behind the VPN.

This was wrapped in a specific GenServer that reran the check every minute. Having an Application and a Supervisor allows this to be started using iex -S mix.

This solved the problem but did require an iex session to be running. The next step was to use a Release to create something that could be run in the background. This only needed a simple `mix release`.

At this point I now had a service that can run as a background task. Shutting it down was a little clumsy – I’d need to find the shutdown script or manually kill it. It would be far simpler if there was a web interface so that it can be stopped via this (or a generic curl command).

Adding a trivial web interface was a simple matter of adding cowboy_plug to the application and configuring some routes. This allows the application to be cleanly shutdown using System.stop/0. Wrapping this in a task allows the web request to return a result before the shutdown.

Aside cowboy_plug has recently gained a dependency upon :telemetry, which needs to be added to your extra_applications. This section allows your application to request the optionally started Erlang application be run. Remember in Erlang terminology an application is more like a library rather than the whole project, which is called a release.

Now that I had a project that could do something in the background and notify the user the obvious thing was to make the GenServer more generic so that it can run any job on a schedule. This also allows me to keep the specific endpoints I am hitting outside a public repo.

This is the route that led to writing home_server. It now checks and notifies that the machine can connect to the internet (another intermittent issue) and has a config file that if present will monitor that URL and provide a custom message.

https://github.com/chriseyre2000/home_server

This project is still at an early stage. I plan to improve the config file as the current version is somewhat naive. It also needs a better solution for starting the processes from the config file.

Cynefin (pronounced CAR-Niven) is a framework for understand the world. There are key states that you can be in. Depending upon the state the optimal approach can vary.

You need to be aware of which zone you are in. If you don’t then you are in the central Confusion zone (which is the most dangerous of all).

Here is the paper that defined the Metastable Failure: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf

The definition is a system failure mode that will remain even if the trigger has been removed.

A simple example would be an unlimited linear retry strategy. A naive approach to retries is to repeat the call after a small pause. Under normal load this can help with an occasional problem. Under extreme load when the server is failing due to excessive requests the retry policy make things worse. Assuming that the overload is at 100 requests per second, a one second retry will generate 200 requests in second 2

This is what happens:

A breaking load becomes persistent. This is why retry policies need to have exponential backoff and a means of giving up.

This is a very simple example, more complex systems can have many ways of getting into these problems.

Some years ago I was working on the replacement version of a big product. We had some of the leading clients using the new version while most were still using the old version. There was a need for a new piece of functionality that would eventually be used by both systems but be introduced into the older system first (as clients get what they want).

To make life easier we asked the team building it to build the new utility as a distinct product that would be integrated via an API. This would have made moving it over simply a case of integrating the API into the new system.

We made the mistake of letting the team define the interface.

A year later when we came to integrate it we found the following contract:

interface ITransfer {

string action(string input)

}

Needless to say it was not a simple replacement job. Every call used the same interface and it was internally mapped to whatever we needed to do. Sometimes it was XML other times list of integers.

Lesson learned: Watch out for overly generic contracts, they are as good as having no contract at all.

A colleague posted an article (https://learnosity.com/not-invented-here-syndrome-explained/) about Not Invented Here Syndrome. This is the tendency for a group to prefer things that they have built versus those that others have created.

The contrasting position is the We Must Do This Because Google Are Doing It (https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb). This is the tendency to try to use techniques that have been successfully used in other large organisations.

You need to know the context of a solution and try to see if it is applicable.
Wardley Maps can help here. (https://twitter.com/swardley/status/764147364605100032) Wardley Maps in Elivish.

You don’t need to invent everything yourself (https://www.ted.com/talks/thomas_thwaites_how_i_built_a_toaster_from_scratch?language=en) nor can you typically acquire everything.

At a previous project we used the following to deploy our terraform code: https://github.com/mergermarket/cdflow

The practical advantage of this is the build process simply uses cdflow deploy envname.

This enforces the similarity of environments, differences only exist in the environment files.
This does make the terraform scripts slightly more complex (although frequently names are postfixed by an environment).

Frequently on a mac you need to get a custom app installed. If you can use the appstore or brew it’s great. However some smaller apps are not on these platforms. The mac has great security systems to protect you from this, but the assumption that everything is malicious can prevent the happy path from being easy.

Recently I resorted to using the following:

https://osxdaily.com/2019/02/13/fix-app-damaged-cant-be-opened-trash-error-mac/

This will after a checklist lead to the following command:

xattr -cr /path/to/application.app

This removes the flag that indicates that the code has been downloaded.
Only do this if you trust the code being run (in my case it was an inhouse tool written by the company that I am contracting at).