The Republicans will impeach all future Democrat Presidents whenever they control the House.
The Brexit trade deal negotiations will last all of 2020 and in the end Boris Johnson will agree to a short extension.
Random outpourings of a software developer
The Republicans will impeach all future Democrat Presidents whenever they control the House.
The Brexit trade deal negotiations will last all of 2020 and in the end Boris Johnson will agree to a short extension.
I have been working with a build pipeline using dependabot this year. At the start of 2019 I took over as team lead on a project. My predecessor had enabled dependabot across the teams pool of repositories. After some pruning we have around 60 of these.
Dependabot is an attempt to solve the upgrade cycle for third party dependencies. It will create a pull request for each new version. If you have a build server configured to run tests that you trust then you can get dependabot to auto merge the passing changes.
The year started with a backlog of 360 PRs. Early in the year I spent the first part of each morning ensuring that almost all of the projects have at least a test pipeline..
This started to make an impact on the backlog. We then encountered our first react update. This added 70 PRs overnight.
One pair on my team automated the deployment of the react libraries to our CDN. This eventually helped (we did have a fun problem with the first bad deploy – if you deploy a bad version of a library to a CDN then its painful to correct – some users will have that version cached potentially forever).
We also needed to restart a lot of broken builds. CD is great but sometimes.cloud infrastructure just fails
Dependabot has a helpful interface in that it has an agent that responds to comments on a PR. This allows you to tell it to rebase/recreate/ignore/merge things.
This week we got the PR total down to 12 by the end of one day. The next day Babel released a new version so we jumped to 70 …
We will never win this race but Dependabot does help keep the code upto date. Note we also get updates from other teams in the organisation. One morning during planning we were able to report that a change we had been waiting on had been released, tested and merged in the time we were meeting.
I have had to raise a few issues with Dependabot support and they did fix them quickly. They had not experienced organisations with over a thousand repos before which made the configuration UI slow – this in now much better.
Our setup of Dependabot, Jenkins and Github did not react well to the introduction of Snyk to the system. Snyk tells Github that it is happy with the security of the dependencies. The problem was this was faster than Jenkins telling Github that it had started testing the change! Dependabot then saw a PR with an ok stamp and then merged the change. We had a fun few days reverting broken builds until we tamed that one.
During the year Dependabot was bought by GitHub and is now free to use.
For now I have disabled Snyk on our projects but will investigate it further in the new year (it appears less configurable than Dependabot).
Here’s to another year of upto date dependencies.
Tea requires boiling water (100C) to brew properly. Giving me a pot of hot water and a tea bag will not allow me to make tea.
I currently work with a project that is controlled by a configuration file.
It has become useful recently to write unit tests for the config file.
We check simple things such as duplicate and even validate keys against a public api.
Given that we are integrating with over 100 services a mistake can cause a large amount of errors. Unit tests help here.
I gave a talk last night at London Functional (hosted by Funding Circle) on Exercism.io and Elixir. During this I attempted a live coding demo.
This had been practiced earlier in the day. I make the typical mistake of trying to make a last min change without retesting. One part of the demo failed. Lesson for the future – don’t change a working demo on the day of a talk!
This looks to be useful:
https://github.com/jan-molak/serenity-js/blob/master/docs/screenplay-pattern.md
I have been mentoring Elixir on Exercim.io for over a year now.
In that time I have mentored 4903 solutions across Elixir and Groovy.
This is a great way to keep your skills upto date in a language.
The students will challenge you with details that you will need to research. It’s the questions that you get that will stretch your knowledge.
You will need to explain unusual bugs and concepts to people who may not have english as their first language.
I have been working with a number of Node projects recently.
Keeping dependencies upto date is a big time sink. I use Dependabot to help with these.
Here is a utility that I have written that allows visualisation of module dependencies: https://github.com/chriseyre2000/package_compare
It loads node_modules into a Neo4j graph database.
To use this you need to install neo4j, create a database user with a password and the Erlang OTP runtime.
Here are the important details:
Once you have run mix escript.build then you can use the following:
./package_compare path-to-the/package.json localhost neo4j_username neo4j_password
This can be run across multiple projects to compare the dependencies. Once you have loaded multiple applications you can you the simple query:
MATCH (a) RETURN a
This will allow you to find the core set of dependencies that your applications are using. If two projects have a large core then there may be a common library waiting to be extracted.
This is an example of an Elixir escript application. This takes an unusual approach with the Sips library, it uses start link itself so that the database configuration can be supplied on the command line. Normally this would be started as a dependent application and the config found from a config file.
The UK is in the middle of a program to implement Smart Motorways on various M and A roads. The intent of these is to allow the hard shoulder to be used as a normal lane except in the event of an accident whereby the variable speed limit signs will mark the lane as out of operation. This seems like a smart idea until you realise the competence of the current variable speed limits.
Variable speed limits can only work if the traffic is capable of travelling faster than the proscribed speed. They can only slow the traffic down. If the variable speed limit is set above the current speed of the road then it is wasting it’s time. This is the majority of uses of the variable speed limits. In addition these limits are kept in place far longer than the problem exists. I have frequently travelled through speed restrictions on the M25 where no broken down car or lose animal was visible. I call the problems these cause artificial traffic jams.
We have an organisation controlling the speeds on motorways that seems unable to reliably determine that a problem has been resolved. Given that this is a similar problem to detection when a breakdown has happen we will have no hard shoulders and traffic just breaking down on the roads.
My team has recently completed a migration (or retirement) of 8 Mongo Databases.
Four of them were replaced with lambda functions. (see https://devrants.blog/2019/05/15/replacing-a-mongodb-with-a-lambda/).
One of the databases had no data to migrate (the data was transitory and had no value after used).
The last three required the used of the Database Migration Service (DMS). DMS is configured to follow a database and move over any updates to the new system. This allows for minimal downtime when moving from one database to another, especially when taking backups and restoring would be prohibitive.
The DMS is very quick to use (we had a slightly slower approach as we had to use Terraform to configure it and could only start the jobs using a Jenkins task).
One flaw we found was in the error handling. One of our databases (unknown to us) contained some text fields with the null character (\u0000). This is something that MongoDB can handle that DocumentDB cannot. The migrations failed, reporting the error, but gave no clue as to where to find the problem. This was problematic as the system we are migrating has around 2 million documents.
We eventually took a brute force approach to find the problem records:
Eventually we found the 40 problem records. These were deleted from the source system and manually inserted into the new.
Eventually we found other problems with DocumentDB that prevented us from using it for the final database (we moved it to another MongoDB provider).
We now have no MongoDB’s on the platform that is being decommissioned.