Concurrent Data Processing In Elixir (B4)

Now that the book has reached the sent to print stage I am again working through it.

Here is the repo containing the samples from the first chapter:

https://github.com/chriseyre2000/sender

The first chapter works through the simplest solution to concurrency: Tasks.
It starts with a simple slow example and step by step adds more sophistication until you get to have a Task.Supervisor watching the code.

This is one of the more elegant examples of demoing the power and simplicity of Elixir tasks that I have seen.

My Elixir Education in a Series of Books

I have been collecting a small library of Elixir books.
This is an introduction to the breadth of the topics covered by Elixir.

A lot of these are tutorial style books that need to be worked through to get the benefits.

My first introduction to Elixir was:

This is Seven More Languages In 7 Weeks which covers a range of languages.
This gave a quick overview of a lot of the language.

Next up was the general introduction book:

Introducing Elixir

This is Introducing Elixir (there is a second version) a fairly straight conversion of Introducing Elixir. This is a gentle introduction to the language.

Next was an earlier version of:

Programming Elixir

This is Programming Elixir (I read one of the earlier editions). This is an in-depth exploration of the language.

Next was

Designing for scalability with Erlang/OTP

This is Designing for Scalability with Erlang/OTP. This goes into more depth on the OTP and how to design for scale. It took me several attempts to work through this.

Next was:

Programming Phoenix

This is Programming Phoenix. I have so far made three attempts to work through this one, the first was in the beta, the second when it was finished (I got distracted) and again recently.

This is one that I am still trying to find the time to read:

Craft GraphQL APIs in Elixir with Absinthe

Each time this one makes it to the top of the list I keep finding another book to read ahead of it.

Metaprogramming Elixir

This is Metaprogramming Elixir which gives a deeper understanding of when to use macros (and when not to).

I won a copy of this on a twitter competition:

Phoenix for Rails Developers

This is Phoenix for Rails Developers another more gentle introduction to Phoenix. The contrast with Rails is illustrating, pointing out pain points that Phoenix solves.

The next book is less about the code and more about how to get a project to use Elixir:

Adopting Elixir

This is Adopting Elixir. This covers some case studies of Elixir being used in production environments.

This is another book that explains how to design with Elixir

Functional Web Development, with Elixir, OTP and Phoenix

This is Functional Web Development with Elixir, OTP and Phoenix.

The next one would make a great second book for Elixir:

Designing Elixir Systems with OTP

This is Designing Elixir Systems with OTP. The approach of building Fun Things, with Big, Loud Worker Bees is a great project structuring approach. It explains the layers that should be used to design a great application.

This is another that I have not yet finished reading, but do get a lot out of:

Learn You Some Erlang for Great Good!

This is Learn You Some Erlang for Great Good! It’s a huge book and covers a lot of details about working with Erlang. I have been meaning to create a repo converting the examples in this book into Elixir.

By the same author is:

Property-Based Testing with PropEr, Erlang, and Elixir

This is Property-Based Testing with PropEr, Erlang and Elixir. The book is more biased towards Erlang. However the ideas in it have changed how I unit test things.

Another one of the books that I have not yet finished (I did buy it in beta):

Real-Time Phoenix

This is Real-Time Phoenix. It covers the soft real-time features of Phoenix.

Another one that I recently finished reading:

Genetic Algorithms in Elixir

This is Genetic Algorithms in Elixir. This covers a topic that you would not naturally associate with Elixir. It makes a good case of why Elixir is very good at it (parallel execution can speed these up).

This is another one that I started working through in beta, and have not yet returned to:

Testing Elixir

This is Testing Elixir. It goes into depth about how to get the most out of ExUnit.

This covers one of the tools that is used heavily by Phoenix.

Programming Ecto

This is Programming Ecto. It covers the database interaction code in more detail than the other books. I like that Ecto provides both Migrations and the data access abstractions.

This is my most recent purchase:

Concurrent Data Processing in Elixir

This is Concurrent Data Processing in Elixir. It seems to cover the tools needed for large scale data processing. Not yet started on this one.

The last one in this list is also as yet unread.

Modern CSS with Tailwind

This is Modern CSS with Tailwind. Technically it is not an Elixir book, but does form part of the PETAL stack (Phoenix, Elixir, Tailwind, Alpine, Liveview). I do plan to create an unofficial repo with the examples for this in Phoenix.

Experimenting With Elixir in Docker

I am working on my book Elixir Function Guide. Having noticed that there are a lot of mix tasks that I did not know about (and some that my exercism.io students don’t) then I thought that I would add some details at the start of the book.

I started using mix help to list the available functions:

mix                   # Runs the default task (current: "mix run")
mix app.start         # Starts all registered apps
mix app.tree          # Prints the application tree
mix archive           # Lists installed archives
mix archive.build     # Archives this project into a .ez file
mix archive.install   # Installs an archive locally
mix archive.uninstall # Uninstalls archives
mix clean             # Deletes generated application files
mix cmd               # Executes the given command
mix compile           # Compiles source files
mix credo             # Run code analysis (use `--help` for options)
mix credo.gen.check   # Generate a new custom check for Credo
mix credo.gen.config  # Generate a new config for Credo
mix deps              # Lists dependencies and their status
mix deps.clean        # Deletes the given dependencies' files
mix deps.compile      # Compiles dependencies
mix deps.get          # Gets all out of date dependencies
mix deps.tree         # Prints the dependency tree
mix deps.unlock       # Unlocks the given dependencies
mix deps.update       # Updates the given dependencies
mix do                # Executes the tasks separated by comma
mix escript           # Lists installed escripts
mix escript.build     # Builds an escript for the project
mix escript.install   # Installs an escript locally
mix escript.uninstall # Uninstalls escripts
mix format            # Formats the given files/patterns
mix help              # Prints help information for tasks
mix hex               # Prints Hex help information
mix hex.audit         # Shows retired Hex deps for the current project
mix hex.build         # Builds a new package version locally
mix hex.config        # Reads, updates or deletes local Hex config
mix hex.docs          # Fetches or opens documentation of a package
mix hex.info          # Prints Hex information
mix hex.organization  # Manages Hex.pm organizations
mix hex.outdated      # Shows outdated Hex deps for the current project
mix hex.owner         # Manages Hex package ownership
mix hex.publish       # Publishes a new package version
mix hex.repo          # Manages Hex repositories
mix hex.retire        # Retires a package version
mix hex.search        # Searches for package names
mix hex.user          # Manages your Hex user account
mix loadconfig        # Loads and persists the given configuration
mix local             # Lists local tasks
mix local.hex         # Installs Hex locally
mix local.phx         # Updates the Phoenix project generator locally
mix local.public_keys # Manages public keys
mix local.rebar       # Installs Rebar locally
mix new               # Creates a new Elixir project
mix phx.new           # Creates a new Phoenix v1.4.0 application
mix phx.new.ecto      # Creates a new Ecto project within an umbrella project
mix phx.new.web       # Creates a new Phoenix web project within an umbrella project
mix profile.cprof     # Profiles the given file or expression with cprof
mix profile.eprof     # Profiles the given file or expression with eprof
mix profile.fprof     # Profiles the given file or expression with fprof
mix release           # Assembles a self-contained release
mix release.init      # Generates sample files for releases
mix run               # Starts and runs the current application
mix test              # Runs a project's tests
mix xref              # Prints cross reference information
iex -S mix            # Starts IEx and runs the default task

It was only then that I realised that I have a number of mix extensions installed.

I need to start with what comes out of the box.
At this point I have three choices:

  • Uninstall the extensions (these are usefull .)
  • Manually edit out the extensions (error prone)
  • Use docker

The docker option is appealing as it will allow me to test using different versions of Elixir.

Here is the simple option that allows the use of docker:

docker run -it elixir:1.10 /bin/bash

This downloads the image and runs up the container. Note that this starts with bash, rather than the default iex.

From this mix help gives the more useful list:

mix                   # Runs the default task (current: "mix run")
mix app.start         # Starts all registered apps
mix app.tree          # Prints the application tree
mix archive           # Lists installed archives
mix archive.build     # Archives this project into a .ez file
mix archive.install   # Installs an archive locally
mix archive.uninstall # Uninstalls archives
mix clean             # Deletes generated application files
mix cmd               # Executes the given command
mix compile           # Compiles source files
mix deps              # Lists dependencies and their status
mix deps.clean        # Deletes the given dependencies' files
mix deps.compile      # Compiles dependencies
mix deps.get          # Gets all out of date dependencies
mix deps.tree         # Prints the dependency tree
mix deps.unlock       # Unlocks the given dependencies
mix deps.update       # Updates the given dependencies
mix do                # Executes the tasks separated by comma
mix escript           # Lists installed escripts
mix escript.build     # Builds an escript for the project
mix escript.install   # Installs an escript locally
mix escript.uninstall # Uninstalls escripts
mix format            # Formats the given files/patterns
mix help              # Prints help information for tasks
mix loadconfig        # Loads and persists the given configuration
mix local             # Lists local tasks
mix local.hex         # Installs Hex locally
mix local.public_keys # Manages public keys
mix local.rebar       # Installs Rebar locally
mix new               # Creates a new Elixir project
mix profile.cprof     # Profiles the given file or expression with cprof
mix profile.eprof     # Profiles the given file or expression with eprof
mix profile.fprof     # Profiles the given file or expression with fprof
mix release           # Assembles a self-contained release
mix release.init      # Generates sample files for releases
mix run               # Starts and runs the current application
mix test              # Runs a project's tests
mix xref              # Prints cross reference information
iex -S mix            # Starts IEx and runs the default task

Funny Characters in Elixir: Sigils

There are a number of special character sequences in Elixir, for example the pipeline operator |> but now I want to discuss the ~something options.

These are sigils. The magic is simple if you call ~x(“something”) then the compiler will translate this into a call to sigil_x

~r/foo/i becomes sigil_r(<<"foo">>, 'i')

See https://elixir-lang.org/getting-started/sigils.html for more details.

This does mean that you can now write your own sigils.

Here are the built in sigils. If there is an uppercase/lowercase pair they work the same except the lowercase one escapes the content.

sigil_C/2          sigil_c/2 returns a charlist

sigil_D/2          Creates date types

sigil_N/2         Creates native datetime

sigil_R/2          sigil_r/2        Regular expressions  

sigil_S/2          sigil_s/2          Creates strings from literals

sigil_T/2          Creates time types

sigil_U/2          Creates a UTC Datetime

sigil_W/2          sigil_w/2          Creates a list of words by splitting on whitespace

         

                  

Announcing About

Last weekend I wrote and released the Elixir package called “about”.

This adds About topics to the iex environment.

It is installed in the usual way:

Add {:about, "~> 0.0.1"} to your deps list in mix.exs

Run mix deps.get

Run iex -S mix

You can now use help topics like:

iex(1)> h About.pipeline

I am looking for suggestions/contributions to the project.

It’s hosted on https://github.com/chriseyre2000/about and available on hex.pm and hexdocs.

Designing Elixir Systems With OTP – Part Three

The B2 edition of this book is out. The new chapter is dedicated to Supervisors. These are correctly described as Lifecycle components.

This is the first time that I have seen the new DynamicSupervisors being used.

Minor note the parameter in

Lifecycle/lib/mastery/boundary/quiz_session.ex for select_session/1 has changed to name from session.

Similarly for answer_question/2

The code samples now clearly state where the code is to be added.

There was a minor typo and some missing code that breaks the final iex session.

Current code can be found at: https://github.com:chriseyre2000/designing_elixir

Working through “Property Based Testing in PropEr with Erlang and Elixir” in Elixir – Part 1

Property testing looks to be an interesting subject.

This is a journal of my working through the book “Property Based Testing in PropEr with Erlang and Elixir”. Note that the book is on it’s third beta at the time I am writing this.

The book is mostly written in Erlang with the Elixir details following (and in an Appendix).

I have upgraded my version of proper to 1.1 rather than the suggested 1.0. This seems to fix the issue I was having experimenting with :proper_types.term()

The PropEr tests will warn you to run mix.propcheck.clean, however if proper is only installed in test mode you will need to use:

MIX_ENV=”test” mix propcheck.clean

The source for this can be found here:

https://github.com/chriseyre2000/pbt

This is at the end of chapter 2.

Property Testing is the practice of testing code with a range of test data that is generated. Typically a property can have 100 test runs with increasingly complex test data. Only the simplest failure will be reported. Properties are a set of invariants that are asserted about the subject under test with respect to the input data.

For example a sorting function can assert that the data set is the same size as before, all elements of each collection (before and after) reside in the other and that each of the sorted elements is <= the next element. Care must be taken not to repeat the implementation. It is possible to use an older system as a golden master.