Script to help review Exercism.io Elixir

~ chriseyre2000 ~ Leave a comment


@rem fix a.test.exs
@rem removes the pending tag from the test and adds _2 to filename
@echo %1 | sed -e "s/.exs/_2.exs/" > temp.txt
@set /p Filename=<temp.txt
@cat %1 | grep -v "@tag :pending" > %Filename%
@rm temp.txt
elixir %Filename%

view raw

fix_test.bat

hosted with ❤ by GitHub

Or on a mac/linux

grep -v @tag *_test.exs > test.exs && elixir test.exs

Contentful to Neo4j in Elixir

~ chriseyre2000 ~ Leave a comment

I just finished an initial port of my neo4j to contentful library from Node to Elixir.

https://github.com/chriseyre2000/contentful_to_neo4j_ex

This is the same utility but in a very different language. It’s not quite as polished as the Node version (but won’t take long to catch up).

Issues that I have had during the port:

  • My machine had a very old erlang implementation (I had installed this when reading the Erlang chapter of 7 languages in 7 weeks – over 7 years ago) which broke HTTPoison, a fairly common HTTP library. The errors pointed to HTTPoison not working on a windows machine (which I have found to not be true).
  • The Contentful Elixir bindings are not very advanced. They don’t return the total number of items that you are paging through. It was not difficult to use the api directly.
  • There are lots of Elixir bindings for Neo4j. Very few of them are clearly documented for writing to Neo4j. Eventually I landed on bolt_sips. Bolt is too primitive, neo4j_sips uses a very old version of HTTPoison.

The Elixir error messages are incredibly clear.

Credo is great for ensuring best practices are applied.

Functional programming allows you to test real code without mocks or spend time fighting promises.

iex is a great REPL environment. You can recompile a module and carry on without restarting everything.

The code is self documenting. This is the top level method:

read_all_assets()
|> read_all_entries
|> process_contentful
|> write_to_neo4j
VSCode is a great editor for elixir. It’s great from the command line
code .
The above will open vscode on the project in the current directory.

Httpoison on windows

~ chriseyre2000 ~ Leave a comment

I have been having trouble getting the Httpoison library working on my windows development machine. I had put this down to portability issues in the past.

Last weekend I spent some time trying to track down the bug so that I could send a patch to the appropriate project.

During this investigation I found that I had been using a fairly old version of Erlang (10) which seemed to work everywhere else but had a problem with the crypto checking that HTTPoison performs. I had installed Erlang when studying 7 languages in 7 weeks over 6 years ago.

Moral of the story: don’t blame the tools, try to fix them and you may find the source of the problem.

Amazon Documentation Also Lies

~ chriseyre2000 ~ Leave a comment

I have spent the last couple of days fighting with oauth 2.

This is based upon Cognito.

The basis of the oauth authentication is a multi step  dance. First you get the user to log in and are redirected back with a code. You need to send back the code with some shared secrets and it responds with a set of tokens.

These tokens do include the refresh token despite the docs saying that they don’t.

The next step is to fetch some user details. This has not yet been implemented yet the docs clearly state that it is there.

I have yet to find the renewal endpoint…

Update…

I have found out why cognito does not implement the ouath/userinfo endpoint.

It’s because the tokens endpoint returns a JWT token that includes the userinfo. I have yet to see if this is the most sane Bearer authentication token.

Testing Promises in Jest

~ chriseyre2000 ~ Leave a comment

I have finally found a decent pattern to test promises.

You can mock a service to return Promise.resolve(x) or Promise.reject(“reason”).

Sample is from contentful-to-neo4j

test(“If contentful is empty then nothing is sent to the db”, (done) => {
  const contentfulService=mockContentfulServiceFactory();
  const neo4jService=mockNeo4jServiceFactory();
  const log=mockLogFactory();
  const systemService=mockSystemServiceFactory();
  contentfulService.getAssets.mockReturnValue(     Promise.resolve(contentfulService.emptyResult));
 contentfulService.getEntries.mockReturnValue(     Promise.resolve(contentfulService.emptyResult));
 const transformService=transformServiceFactory(contentfulService, neo4jService,   contentfulBatchSize, log, systemService);
 transformService.copyContentfulSpaceToNeo4j();
  setTimeout( () => {
     expect(contentfulService.getAssets.mock.calls.length).toEqual(1);
     expect(contentfulService.getEntries.mock.calls.length).toEqual(1);
     expect(neo4jService.cypherCommand.mock.calls.length).toEqual(0);
     done();
    },

  1);
});
The trick is to use setTimeout of 1 to happen after the promises have been resolved.
Since the promises are already complete they all resolve before the next millisecond. The done holds the test up for the timeout.

Sample neo4j queries for the contentful-to-neo4j graph

~ chriseyre2000 ~ Leave a comment

I have now stabilised the contentful-to-neo4j project. It should be able to handle most contentful spaces. I still need to tune the transactions for very large datasets – but have no errors yet.

It has gone from being a simple script to an application that has 100% code coverage. I have learnt a lot about javascript Promises and how to test in Jest.

Here are some useful neo4j queries:

Find out how many of each content type that you have published:

MATCH (n) WHERE n.contenttype IS NOT NULL

RETURN n.contenttype, COUNT(n)

ORDER BY n.contenttype

You need to remember that nodes are circular and need to be surrounded by ()

Find pages that share slugs:

MATCH (a {slug:’common-url’}) RETURN a

Find nodes that have a slug with a specific ending:

MATCH (n) WHERE n.slug ENDS WITH ‘-kr’ RETURN DISTINCT n

Find assets that are not referenced by a content type:

MATCH (image {cmstype: ‘Asset’})
WHERE NOT (image)-[]-()
RETURN image

Simple search for orphans:

MATCH (a) WHERE NOT (a)-[]-() RETURN a

These are simple queries in Neo4j that would be hard to do in the Contentful UI.

Analysing Contentful Spaces in Graphene hosted in Heroku

~ chriseyre2000 ~ Leave a comment

I have just updated my contentful-to-neo4j project so that it will by default work on a graphene database hosted in Heroku.

Steps to get this working:

Clone the repo:

git clone https://github.com/chriseyre2000/contentful-to-neo4j

Sign up to heroku, add a credit card, create an empty heroku app.

Add the free graphene db addon (assuming that you have less than 1000 content types).

Install the heroku cli

Open a terminal in the directory that you checked out the repo.

This may require you to login to heroku on the cli.

heroku git:remote -a YOUR_APP_NAME

set the two keys that identify the contentful space:

SPACE_ID

CONTENTFUL_ACCESS_TOKEN

Once the app has finished restarting:

heroku run loader -a YOUR_APP_NAME

You should now have a full populate graphene database.

You may need to move to a paid tier big enough for your contentful space…

This is surprisingly painless. I only had to add environment variable fallbacks and a Procfile.