Seven Databases in Seven Weeks Part 1

The project that I am working on finally got the requirements that allowed us to pick a database. Over the last few years I have been using MongoDb as the lazy choice for storage. The project makes more sense to use a relational database so I have returned to Postgres.

This has encouraged me to reread this book. I am planning on adding some details in these notes to add modern practices to the book.

To start with the book predates containers so all of the chapters require you to install the database on your machine. I am not encouraging you to use a database in a container for production. It does however allow you to test drive database code.

Over the last decade all of the big databases I have used have been cloud hosted. For a JVM based application we used hsqldb as an embedded replacement for the database in tests.

The book also fails to mention database migrations. This is a technique that allows you to version control the schema of a database.

Coming back to Postgres after years of Mongo makes me realize that you need to work a little harder to get data to and from the store. On the other hand SQL is far easier to make aggregate queries with compared to Postgres. It is easier to enforce constraints in SQL. Types, unique indexes, non-null fields and foreign keys all help.

The book also only works at the database level. I want to add examples of how to connect to the database.

The Trouble With Terraform

Recently I have been working with Terraform to stand up the infrastructure for the project that I am working on. The project involves a database plus a number of lambda functions. We have a CD build pipeline and use Terraform for the infrastructure, run from cdflow.

Terraform is great for defining the infrastructure. The problem comes when you rename something (module names in particular). Terraform will attempt to tear down the service and recreate it. Sometimes AWS services are eventually consistent. This can mean that a deleted resource hangs around for a while after being deleted. A rename of a module will delete and recreate the item, which will frequently fail on the first pass.

You also need to be very careful that you only build to a given environment from a single branch of the build pipeline. Not doing so allows databases to be torn down. I have seen an incident where a developer comments out infra that is not needed for the current build, only for that change to delete the production database and all the backups. There are things that can be done to prevent this (marking backups as requiring an extra switch to be removed). These changes then leave Terraform unable to completely clean up.

I am not arguing against configuration as code, merely noting that you will get a lot of failed builds. Some of these are resolved by rerunning the job/pipeline. Others require the resource to be manually deleted.

Terraform operates at a level of abstraction that can both help and hinder. Some of the abstractions are a little weak (these will improve in time) especially when defining IAM permissions as you end up with inline strings that explicitly define the policy.

Why Dependabot Works

This is the theory of why dependabot works so well.

With atomic changes you know what triggered the break (or find that your tests are unreliable, which is also valuable).

Genetic Algorithms In Elixir

This weekend I have been working through the published version of this book.

The ideas are great, but there are too many errors in the examples to make it easy to work through.

The book builds up a framework for using Genetic Algorithms, including some sophisticated logging, visualisation and performance tuning tricks. Some of the example code used clearly belongs with earlier drafts. I am sure if you download the sample code you could fix the mistakes. For example some of the logic in the tiger example is wrong in the first instance, but correct when used later on. Also I don’t think that you can pass an anonymous function to apply.

Here is the repo that I have been working on:

https://github.com/chriseyre2000/genetic

Identity Theft

Recently I have been the victim of Identity Theft.

In October and November last year there were a total of 57 attempt to take out financial services in my name. So far two of these have succeeded in obtaining money.

Both of them were payday loan companies. I have reported the identity theft to Action Fraud. Both companies would openly talk to me about the loans and have recorded them as fraudulent.

I have signed up to a credit file checking service to allow me to see what the current state is. I will need to keep this up for a couple of months.

However neither of them have made any attempt to check that I am who I say I am. I have been sent the pdf bank statement used as part of the identification by one of the companies. Other than giving them publicly available information about myself (name, date of birth and address) no attempt has been made to validate who I am.

It can be hard to get a company to talk to you about fraud. Firms should really have an email address that can be used to report this. For example one of the finance checks went to Sky Mobile. I cannot find a means of talking to them without becoming a customer.

GDPR requests seem to be the only way to get a company to talk to them.

Programming Phoenix Chapter 12

This chapter adds an OTP application to the demo.

This is the first chapter where I have found some typos
info_sys/application.ex needs the following to work:

alias InfoSys.Counter

Without that the app won’t start.

You also need to be careful with the examples. The name of the files are sometimes incorrect (although the listed path is right). Sometimes it asks you to edit a file in Rumbl when it actually means InfoSys.

There are also a lot of mistakes in the supervisor demos. Sometimes aliases are missed (as above) and sometimes it refers to the wrong project.

Again working though Programming Phoenix (part 2)

I have continued working through the Programming Phoenix book.

https://github.com/chriseyre2000/rumbl

Currently I am at the end of Chapter 10.
This has introduced using websockets in Phoenix to interact with the page.
The application allows annotations to be made to a video that playback in real time.

I am having some trouble with the actual Youtube API since it now complains about cookies not being on the main site. I intend to finish working through the book and return to fix this issue.

Chapter 11 is about refactoring the application to use an Umbrella Application. This will split the Rumbl and RumblWeb applications apart. Having a real demo of refactoring is useful.