Replacing a MongoDB with a Lambda

The project that I currently work on uses a number of microservices. Several of these microservices use MongoDB. The platform provider that we are using to host these databases ( has announced that they are going to stop supporting Mongo in about two months time.

This has resulted in a thorough investigation of the mongo databases that we are using.

Several of the services use the following architecture:

This has three instances of an application that periodically poll a website before performing sending the result to rabbitmq. They use the mongodb as a lock so that only one of the three perform the operation.

The alternative design was to add an external scheduler to trigger the nodes. The load balancer would pick one of the nodes to perform the work. One of my colleagues pointed out that we could use a lambda to do this. Lambdas are (typically) javascript functions that are cloud hosted.

It it is possible to deploy a lambda with a schedule of how frequently it should be invoked, either with a given frequency or via a CRON-like specification. The lambdas invoke methods on the nodes that asynchronously trigger the process.

Here is a diagram of the replacement:

This has allowed us to replace three MongoDB’s in each of four environments with a much simpler lambda function.

In addition we have been able to remove the mongodb database driver and the internal cron scheduling components. This greatly simplifies these services. There is now less to maintain (less PR’s from dependabot). It is also far easier to test code that is called from an endpoint than one from a scheduled event.

We do still need to replace the remaining MongoDB’s which contain real data but this is a much more manageable job.

We also learned that AWS still uses Node 8 LTS for Lambda.

This is a great starting point for serverless functions.

Contentful to Neo4j in Elixir

I just finished an initial port of my neo4j to contentful library from Node to Elixir.

This is the same utility but in a very different language. It’s not quite as polished as the Node version (but won’t take long to catch up).

Issues that I have had during the port:

  • My machine had a very old erlang implementation (I had installed this when reading the Erlang chapter of 7 languages in 7 weeks – over 7 years ago) which broke HTTPoison, a fairly common HTTP library. The errors pointed to HTTPoison not working on a windows machine (which I have found to not be true).
  • The Contentful Elixir bindings are not very advanced. They don’t return the total number of items that you are paging through. It was not difficult to use the api directly.
  • There are lots of Elixir bindings for Neo4j. Very few of them are clearly documented for writing to Neo4j. Eventually I landed on bolt_sips. Bolt is too primitive, neo4j_sips uses a very old version of HTTPoison.

The Elixir error messages are incredibly clear.

Credo is great for ensuring best practices are applied.

Functional programming allows you to test real code without mocks or spend time fighting promises.

iex is a great REPL environment. You can recompile a module and carry on without restarting everything.

The code is self documenting. This is the top level method:

|> read_all_entries
|> process_contentful
|> write_to_neo4j
VSCode is a great editor for elixir. It’s great from the command line
code .
The above will open vscode on the project in the current directory.