Viewing a Contentful space in Neo4j part 2

I have now completed a minimal version of the mapping tool.

The code can be found in this github repo:

https://github.com/chriseyre2000/contentful-to-neo4j

There are some warnings before you use this script:

  • It will empty the target neo4j database before populating it.
  • All of the data is written to neo4j in a single transaction. This will be corrected soon.
  • I don’t yet handle all primitive field types.
  • I have not yet added links to assets embedded in markdown fields.
  • It has only been tested on a local machine with a local neo4j.

I can’t guarantee that it will work with all Contentful spaces yet. So far I have only tested it on the demo space that comes when you sign up for a free Contentful account.

If you do have problems please send me either a pull request or a failing test case (show me the schema of the problem content type).

Once you have the neo4j database populated it becomes trivial to find orphaned items. I’ll add some useful samples queries to the documentation.

API vs Library

In my previous post I mentioned that I am trying to write to Neo4j from node.

This is becoming difficult as each of the top two libraries seems to have serious dependency problems.

The Neo4j package does not handle error conditions well due to a missing stacktrace function.

The neo4j-driver package is also broken with a vague “Headers is not defined”.

This brings me to the main topic: when should you choose a library versus directly using an api?

Firstly does the library do the job? There are lot of node packages out there. Some of them are useful.

If the library adds some features then it’s a no-brainer. Especially if the library can maintain a stable contract despite the underlying api changing (The Delphi VCL was a great example of this. The VCL survived the underlying platform moving from 16 to 32 bit without any code changes required).

There is also the matter of the dependency chain that you will pick up with the library. It’s not uncommon for a npm package to have tens of dependencies. This can mean that a poorly maintained library can be broken by someone else’s change (this may be less of a problem with the introduction of package-lock). This can also be difficult with dependencies from multiple required libraries: these can clash and you may have to chose which bugs you can live with.

If the usage that you have of an api is simple and the api is stable it may be worth directly calling just the parts that you need.

Viewing a Contentful space in Neo4j – Part 1

Contentful is a really effective headless CMS. It’s api does have some limits (you can only query user defined fields across a single type at a time).

A few years ago I managed to find a way of mapping a contentful space into a Neo4j graph database. This allows full querying of the data in contentful. This can be useful for finding where a given image is in use or finding pages that are orphaned.

I am now trying to recreate this library as an open source node project.

I have started by creating a free contentful account (provided I only have one space and live within the limits this will be fine for my purposes).

To query contentful I am using the contentful npm package.

So far I can query the assets and content types in my space.

Initially I am going to use a local version of neo4j but will be moving to a Heroku hosted version. Neo4j community 1.1.6

I’ll add to this series as I progress.

Testing React with Enzyme, Jest and Fetch

I have been having fun recently working on a react app. Typically I use TDD.

This can be difficult when you have code that goes:

fetch.then(res => json)
.then(res => this.setState(res.aValue)
 )

Testing this gets awkward. Changing the signature is too invasive. You can easily mock the fetch but get a race condition reading the state.

This is the solution that I found works. By abusing the javascript sequencing a bit – do what you need to then check this:

setTimeout(() => { assert thing.state}, 0)

Thoughts On Contentful Migrations

Contententful have a wonderful headless CMS. It has a UI for the content editors but the developers get a stream of JSON. This means that the consuming application is free to do anything with the output – which is a major contrast to other CMS systems.

Contentful have a CLI tool that allows migrations to be applied to the schemas (and content). This is not the approach that my team has taken (largely because the migration tool did not exist when we needed it).

Our approach is to have a declarative content schema in code. On deployment it compares this to the environments current state and attempts to upgrade it.

Upgrades happen on a field level of each content type. Once a content type has been upgraded then the widget type for each field is set.

We don’t attempt to remove fields or migrate data as we have not in two years had the need to automate this (we have manually removed a few fields or redundant content types).

This approach is closer to desired state configuration than migrations. It has the benefit of acting as it’s own complete documentation.