Announcing About

Last weekend I wrote and released the Elixir package called “about”.

This adds About topics to the iex environment.

It is installed in the usual way:

Add {:about, "~> 0.0.1"} to your deps list in mix.exs

Run mix deps.get

Run iex -S mix

You can now use help topics like:

iex(1)> h About.pipeline

I am looking for suggestions/contributions to the project.

It’s hosted on https://github.com/chriseyre2000/about and available on hex.pm and hexdocs.

Designing Elixir Systems With OTP – Part Three

The B2 edition of this book is out. The new chapter is dedicated to Supervisors. These are correctly described as Lifecycle components.

This is the first time that I have seen the new DynamicSupervisors being used.

Minor note the parameter in

Lifecycle/lib/mastery/boundary/quiz_session.ex for select_session/1 has changed to name from session.

Similarly for answer_question/2

The code samples now clearly state where the code is to be added.

There was a minor typo and some missing code that breaks the final iex session.

Current code can be found at: https://github.com:chriseyre2000/designing_elixir

Working through “Property Based Testing in PropEr with Erlang and Elixir” in Elixir – Part 1

Property testing looks to be an interesting subject.

This is a journal of my working through the book “Property Based Testing in PropEr with Erlang and Elixir”. Note that the book is on it’s third beta at the time I am writing this.

The book is mostly written in Erlang with the Elixir details following (and in an Appendix).

I have upgraded my version of proper to 1.1 rather than the suggested 1.0. This seems to fix the issue I was having experimenting with :proper_types.term()

The PropEr tests will warn you to run mix.propcheck.clean, however if proper is only installed in test mode you will need to use:

MIX_ENV=”test” mix propcheck.clean

The source for this can be found here:

https://github.com/chriseyre2000/pbt

This is at the end of chapter 2.

Property Testing is the practice of testing code with a range of test data that is generated. Typically a property can have 100 test runs with increasingly complex test data. Only the simplest failure will be reported. Properties are a set of invariants that are asserted about the subject under test with respect to the input data.

For example a sorting function can assert that the data set is the same size as before, all elements of each collection (before and after) reside in the other and that each of the sorted elements is <= the next element. Care must be taken not to repeat the implementation. It is possible to use an older system as a golden master.

Software Reliability

One of my colleagues asked at a recent open spaces event whether the software industry was doing enough about Software Reliability. This was inspired by Bob Martins talk: https://www.youtube.com/watch?v=17vTLSkXTOo This raised the fear of 10K deaths caused by software and a potential future restrictions.

To start with I mentioned the aircraft practice of having three independent teams write distinct solutions that need to have two of them agree for any set of input.

The next item is the Erlang/Elixir/OTP ecosystem with its supervisor trees. This is the Erlang principle of “let it crash”. Erlang was designed to allow the software to fail and expect the machine it runs on to fail. This is why the Erlang VM is designed to be distributed – it’s the only way to protect against failure. It even allows software to be upgraded while running. This the software system that runs: telephone switches, Heroku, Rabbit MQ and Whatsapp.

Then there are tools that can help reliability:

Saboteur (https://github.com/tomakehurst/saboteur) is a tool that can inject network failures between parts of the system. This allows delays and blocks to be simulated. Systems can be tested for resilience – how they behave when the network fails and then recovers.

Gatling (https://gatling.io/) is a load test tool. This allows us to see how a system reacts under load. One place that I worked tested to either three times the peak load or to system failure. This involved having twelve instance of the application installed in aws vm’s around the world. Some or all of these could be pointed at a system with a suite of scenarios. This would have 300 users arriving per second (and then use the system) for 2 hours. A good test run would involve the system still being responsive throughout this load and then cleanly recovering afterwards.

Property Testing (https://github.com/proper-testing/proper or http://hackage.haskell.org/package/QuickCheck). These are tools that allow systems to be tested against generators that try to examine behaviour against the entire of the parameter space. Suites of random values are tested and upon a failure it attempts to find the simplest example that causes the same problem. This is documented here: https://pragprog.com/book/fhproper/property-based-testing-with-proper-erlang-and-elixir . Note that that book is still in beta.

There are the resources out there, it is up to the software development community to use them to raise their game.

 

Elixir Supervisor Introduction

This is a simple introduction to OTP in Elixir.

The sample is based heavily upon Introducing Elixir.

This will demonstrate how Elixir keeps a server alive across code recompilation, and restores a server after it has crashed.

This assumes that you have installed Elixir.

On a mac you can install this with (there are other solutions on other platforms):

brew install elixir

Start by cloning the repo:

git clone https://github.com/chriseyre2000/drop_server.git

Change directory into the drop_server folder.

Type

iex -S mix

This will start the application running inside the interactive Elixir REPL.

This application has a DropServer.Worker being monitored by a supervisor.

First you can calculate the velocity after falling a distance in meters:

(Type the bit after the iex> prompts):

iex(1)> DropServer.Worker.calculate_drop(40)

{“ok”, 28.0}

iex(2)> DropServer.Worker.calculate_drop(41)

{“ok”, 28.347839423843222}

iex(3)> DropServer.Worker.calculate_drop(42)

{“ok”, 28.691462144686877}

iex(4)> DropServer.Worker.how_many_calls

So far calculated 3 velocities.

:ok

You can even recompile the DropServer.Worker while it is running:

iex(5)> c(“lib/drop_server/drop_server.ex”)

warning: redefining module DropServer.Worker (current version loaded from _build/dev/lib/drop_server/ebin/Elixir.DropServer.Worker.beam)

lib/drop_server/drop_server.ex:1

warning: redefining module DropServer.Worker.State (current version loaded from _build/dev/lib/drop_server/ebin/Elixir.DropServer.Worker.State.beam)

lib/drop_server/drop_server.ex:4

[DropServer.Worker, DropServer.Worker.State]

This even maintains the state:

iex(6)> DropServer.Worker.how_many_calls

So far calculated 3 velocities.

:ok

Now if you give it some invalid data Elixir will do the classic Erlang thing and let it crash!

iex(7)> DropServer.Worker.calculate_drop(-1)

21:49:21.278 [error] GenServer DropServer.Worker terminating

** (ArithmeticError) bad argument in arithmetic expression

(stdlib) :math.sqrt(-19.6)

(drop_server) lib/drop_server/drop_server.ex:44: DropServer.Worker.fall_velocity/1

(drop_server) lib/drop_server/drop_server.ex:20: DropServer.Worker.handle_call/3

(stdlib) gen_server.erl:661: :gen_server.try_handle_call/4

(stdlib) gen_server.erl:690: :gen_server.handle_msg/6

(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

Last message (from #PID<0.133.0>): -1

State: %DropServer.Worker.State{count: 3}

Client #PID<0.133.0> is alive

(stdlib) gen.erl:169: :gen.do_call/4

(elixir) lib/gen_server.ex:921: GenServer.call/3

(stdlib) erl_eval.erl:680: :erl_eval.do_apply/6

(elixir) src/elixir.erl:265: :elixir.eval_forms/4

(iex) lib/iex/evaluator.ex:249: IEx.Evaluator.handle_eval/5

(iex) lib/iex/evaluator.ex:229: IEx.Evaluator.do_eval/3

(iex) lib/iex/evaluator.ex:207: IEx.Evaluator.eval/3

(iex) lib/iex/evaluator.ex:94: IEx.Evaluator.loop/1

** (exit) exited in: GenServer.call(DropServer.Worker, -1, 5000)

** (EXIT) an exception was raised:

** (ArithmeticError) bad argument in arithmetic expression

(stdlib) :math.sqrt(-19.6)

(drop_server) lib/drop_server/drop_server.ex:44: DropServer.Worker.fall_velocity/1

(drop_server) lib/drop_server/drop_server.ex:20: DropServer.Worker.handle_call/3

(stdlib) gen_server.erl:661: :gen_server.try_handle_call/4

(stdlib) gen_server.erl:690: :gen_server.handle_msg/6

(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

(elixir) lib/gen_server.ex:924: GenServer.call/3

However only the state is lost:

iex(7)> DropServer.Worker.how_many_calls

So far calculated 0 velocities.

:ok

iex(8)> DropServer.Worker.calculate_drop(10)

{“ok”, 14.0}

iex(9)> DropServer.Worker.how_many_calls

So far calculated 1 velocities.

:ok

The supervisor has restarted the server.

In a real service the state would have been kept in a distinct service possibly with some form of persistence.

Elixir Metaprogramming : the basics

Elixir has a very small core language. Most of what is thought of as the syntax is actually written using macros.

Elixir makes it ridiculously easy to get at the Abstract Syntax Tree (AST) of the code that you are using.

How to get the AST of some code:

iex(1)> quote do: 1 + 1
{:+, [context: Elixir, import: Kernel], [1, 1]}

Contentful to Neo4j in Elixir

I just finished an initial port of my neo4j to contentful library from Node to Elixir.

https://github.com/chriseyre2000/contentful_to_neo4j_ex

This is the same utility but in a very different language. It’s not quite as polished as the Node version (but won’t take long to catch up).

Issues that I have had during the port:

  • My machine had a very old erlang implementation (I had installed this when reading the Erlang chapter of 7 languages in 7 weeks – over 7 years ago) which broke HTTPoison, a fairly common HTTP library. The errors pointed to HTTPoison not working on a windows machine (which I have found to not be true).
  • The Contentful Elixir bindings are not very advanced. They don’t return the total number of items that you are paging through. It was not difficult to use the api directly.
  • There are lots of Elixir bindings for Neo4j. Very few of them are clearly documented for writing to Neo4j. Eventually I landed on bolt_sips. Bolt is too primitive, neo4j_sips uses a very old version of HTTPoison.

The Elixir error messages are incredibly clear.

Credo is great for ensuring best practices are applied.

Functional programming allows you to test real code without mocks or spend time fighting promises.

iex is a great REPL environment. You can recompile a module and carry on without restarting everything.

The code is self documenting. This is the top level method:

read_all_assets()
|> read_all_entries
|> process_contentful
|> write_to_neo4j
VSCode is a great editor for elixir. It’s great from the command line
code .
The above will open vscode on the project in the current directory.