Co·sent: shared perception; joint knowledge; collective intelligence.
How to run continuous integration on private Plone projects, for free.
Continuous integration testing is a key enabling practice for agile coding projects. In the Plone community, it's become best practice to combine a Github code repository with a Travis-CI testing integration for separate packages. In addition, the Plone CI team uses Jenkins to test the overall Plone integration.
Both Github and Travis are free for public open source projects, but running private projects on Travis is an expensive proposition. Jenkins can be hard to set up and maintain, and the user interface is becoming long in the tooth.
Enter Gitlab and Gitlab-CI. Gitlab shamelessly replicates most of the Github user experience. But, in contrast to Github, private repositories are for free on Gitlab. In addition, Gitlab-CI offers Continuous Integration features for free also.
For a new Quaive customer project, I needed a private repository with continuous integration testing. Below I'm sharing with you how you can set up your private Plone projects on Gitlab-CI, too.
These are the steps involved to make this work:
- Create an account on Gitlab.
- Create a repo and push your current code base.
- Prepare a Docker image to use as a base for Gitlab-CI testing.
- Prepare your code for Gitlab-CI testing.
- Configure, start and attach Gitlab-CI test runners.
- Push your work on any branch and check test results.
You can figure out the first two steps on your own.
If you want to use Gitlab as a second repository and pipeline, in addition to e.g. an existing Github origin, you can simply add another remote and push there:
git remote add gitlab <url> git push gitlab master
If you're still afraid of Docker, now is the time to overcome that fear. The way Gitlab-CI uses docker is awesome and a gamechanger for testing. Here's the benefits I'm enjoying:
Full isolation between test builds and test runners
We've been wrestling with cross-build pollution on Jenkins, especially with the Github pull request builder plugin. No such thing on Gitlab-CI, every test run is done in a new, pristine Docker container which gets discarded after the run.
Fully primed cache for fastest buildout possible
This was the key thing to figure out. On Travis the buildout is typically primed by downloading the Plone unified installer, extracting the eggs cache from the tar ball and using that to speed up the buildout. That still leaves you with running that download plus any extra eggs you need, for every build. Not so on Gitlab-CI. I'll explain how to set up your own Docker cache below.
Unlimited, easily scalable test runners
The Gitlab-CI controller is for free. The actual test runs need slaves to run your builds. What is super neat about Gitlab-CI, is that your test runners can run anywhere. Even on your laptop. That means you don't have to pay for extra server capacity in the cloud: you can just use your desktop, or any machine with spare capacity you have, to run the tests.
Imagine what this does during a sprint. Normally, CI capacity can quickly become a bottleneck if lots of developers are pushing a lot of code on the same days. Because of the scalable nature of Gitlab-CI, every developer could just add a runner on his laptop. Or you could hire a few cheap virtuals somewhere for a few day to temporarily quadruple your testing capacity for the duration of the sprint.
Parallel test runs for faster end results
As a result of all of the above (fast buildouts, scalable runners) it becomes feasible to split a long-running test suite into several partial runs, you then run in parallel. At Quaive, a full test run with nearly 900 tests takes the better part of an hour to run. Being able to split that into two shorter runs with faster feedback on failures is a boon.
Prepare a Docker image to use as a base for Gitlab-CI testing
You have to, once, prepare a Docker image with all system dependencies and an eggs cache. That image will be cached locally on your CI runners and used as a base for every test run.
I use a Docker image with a fully primed cache:
The Dockerfile pulls in all system dependencies and runs the buildout once, on creation of the Docker image.
docker build -t yourorganisation/yourproject
The buildout pulls in all the needed eggs and download sources and stores them in a buildout cache. Note that in case of complex buildout inheritance trees the simplest thing to do is to just list all eggs alphabetically, like I've done here.
Once that's done, you create an account on Docker hub and push your image there:
docker push yourorganisation/yourproject
Note that Quaive is quite a complex code base. For a less complex Plone project you could prune the list of installed system packages and eggs. YMMV.
Prepare your code for Gitlab-CI testing
You simply add a .gitlab.yml script that configures the test runs. This is required and the file must have that name.
In addition, it makes sense to add a specialized gitlab.cfg buildout file, but this is not required and the name is completely free.
For those of you who have worked with .travis.yml before, this will look very similar:
before_script: - export LC_CTYPE=en_US.UTF-8 - export LC_ALL=en_US.UTF-8 - export LANG=en_US.UTF-8 - virtualenv --clear -p python2.7 . - bin/pip install -r requirements.txt - bin/buildout -c gitlab-ci.cfg - /etc/init.d/redis-server start - Xvfb :99 1>/dev/null 2>&1 & robot: script: - DISPLAY=:99 bin/test -t 'robot' norobot: script: - bin/code-analysis - bin/test -t '!robot' - ASYNC_ENABLED=true bin/test -s ploneintranet.async
Here's what's happening:
before_script configures the system to be UTF-8 safe, bootstraps the buildout and then builds out a gitlab-ci.cfg which is just a normal buildout, but stripped from everything not needed in the test. We then start any required services, in our case we need redis. A Xvfb virtual framebuffer is needed for the robot tests.
This before_script is always run, the rest defines separate CI runners:
robot runs our robot tests on the virtual framebuffer we just started.
norobot runs the rest of the tests, plus an extra test run of our async stack which does not mock async but actually exercises celery and redis.
So this defines two CI runners, with a shared setup fixture that is used in both.
Configure, start and attach Gitlab-CI test runners
If you followed the above closely, you'll notice that the Docker image we configured has not been referenced yet. That follows in this step, which brings everything together.
On your Linux workstation or server you want to use as a host for running the CI tests, first install Docker:
sudo apt-get install docker.io
Then, start a runner in auto-reboot mode:
docker run -d --name gitlab-runner-01 --restart always \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /srv/gitlab-runner-01/config:/etc/gitlab-runner \ gitlab/gitlab-runner:latest
If you want to have multiple runners in parallel, repeat the above, replacing gitlab-runner-01 with gitlab-runner-02 etc.
Now go to your Gitlab account. Navigate to your project. In the sidebar, choose "Settings". Scroll down the sidebar and choose "Runners".
First disable the shared runners provided by Gitlab. They're not suitable for testing your Plone project.
Now, while keeping this screen open (You'll need to copy the registration token), go to your Linux terminal again and register the runner you started above, as a runner for this Gitlab project:
docker exec -it gitlab-runner-01 gitlab-runner register
This will involve you in something like the following dialog:
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/ci ) https://gitlab.com/ci Please enter the gitlab-ci token for this runner xxx copy from Gitlab settings xxx Please enter the gitlab-ci description for this runner myhost-runner-01 INFO fcf5c619 Registering runner... succeeded Please enter the executor: shell, docker, docker-ssh, ssh? docker Please enter the Docker image (eg. ruby:2.1): yourorganisation/yourproject:latest INFO Runner registered successfully.
Here in the last question you use the name of the Docker image you pushed to Docker hub before.
Repeat the registration process in case you want to add multiple runners.
That concludes the hard part.
Push your work on any branch and check test results
Now, any push to any branch on your Gitlab project will trigger the builds you configured. You can see them in the sidebar of your project under "builds" and you can follow the console of the build as it's progressing.
What I like about this build system is the "Retry build" button. No more pleading with Jenkins on Github comments to try and trigger the pull request builder (which happens to ignore the trigger phrases you configured because it always only uses it's own hardcoded triggers).
Also, you don't need to open a fake pull request just to trigger a build. So annoying to open a pull request overview on Github and see lots of outdated, broken builds which are not really pull requests but just a hack to get Jenkins going. No more.
There's two gotchas you need to be aware of here:
Docker may not have been started properly.
There's a known race condition between the docker service and your firewall which prevents your dockers from running properly. That shows up as "hanging" builds in your build overview. The solution is simple: just once after reboot, issue a sudo /etc/init.d/docker restart and that should fix it.
Branch testing on Gitlab is subtly different from pull request testing on Github.
Gitlab is very straightforward: it tests your branch at the last commit you pushed there. The downside of that is, that if your branch is behind master, a merge may result in a broken master, even though the branch itself was green. The way to prevent that is to allow only fast-forward merges, which you can configure. Personally I have reasons to not always fast-forward. YMMV.
Github on the other hand, tests your pull request after virtually merging to master. The upside of that is clearly, that a green PR indicates that it can be safely merged. The downside is more subtle. First off, not many developers are aware of this virtual merging. So if your branch is behind master it may break for a regression which you cannot reproduce on your branch because it needs a merge/rebase with master first. So you can have an unexplicable failure. The other is also possible: you may have a green test result on your PR but still breakage on master, if some other PR got merged in the meantime and you did not trigger a new test on the to-be-merged PR.
On Quaive, we now use Jenkins-CI with Github pull request integration, in parallel with the Gitlab-CI setup described above. This makes it very easy to triangulate whether a test failure is caused by code or by the test server. It also provides us with both the benefits of per-branch per-push Gitlab-CI testing and the virtual merge pull request testing of Gitlab + Jenkins.
If you're interested in setting this up for your own projects and running into issues, just start a thread on community.plone.org and ping me (@gyst), or leave a comment below this post.
How social intranet design can enable organisational change.
This talk, presented at IntraTeam Event 2016, introduces a four-type classification of organisational cultures, based on the Cubrix model of organisational development.
Using the example of the iKath intranet of the Swiss Catholic Church in the canton of Zurich, each of these cultural types shows up as a set of related functionalities in the intranet design.
The layered interaction design is adaptive: users can safely ignore the "social layer" and still have a fully functional user experience that addresses document management and process support. This enables a gradual evolution of organisational change, without requiring retooling.
Collaborating with nine companies across five countries, the Plone Intranet Consortium moved from plan to working code within months. How did we do it?
Fast-forward to summer 2015: we've already done a Mercury "technology preview" release and are now in feature freeze, preparing for the 1.0 release, codename Venus, this summer.
As you can see in the video, for all of us it's very important to be part of the open source community that is Plone.
At the same time, we use a different process: design driven, which impacts our code structure and the way integrators can leverage the Plone Intranet platform.
Sharing and re-use
All of our code is open source and available on Github. In terms of re-use we have a mixed strategy:
First of all it's important to realize we're doing a design-driven product, not a framework. We have a vision and many components are closely integrated in the user experience (UX). From a UX perspective, all of Plone Intranet is an integrated experience. Sure you can customize that but you have to customize holistically. You cannot rip out a single feature and expect the UX for that to stand on it's own.
In the backend the situation is completely different. All the constituent packages are separate even if they live in one repo and one egg. You can install ploneintranet.microblog without installing the whole ploneintranet stack: i.e. the whole ploneintranet source needs to be there (at the python level) but you can load only the ploneintranet.microblog ZCML and GS and you'll be fine. All our packages have their own test suites which are run independently. Of course you need activitystream views to display the microblog - and that's frontend UX and one of the most complex and integrated parts of our stack, with AJAX injections, mentions, tagging, content mirroring and file preview generation.
Another example is search: a completely re-usable backend but you'd have to provide your own frontend. Our backend is pluggable - we currently support both ZCatalog and Solr engines and expect to also support Elastic Search in the future. We have documented our reasons for not reusing collective.solr.
Design and user experience are key
We don't believe that loosely coupled components with independent developer-generated frontends create a compelling user experience. Instead of working from the backend towards the frontend, we work the other way around and focus on creating a fully integrated, beautiful user experience.
The downside of that is that it becomes more difficult to reuse components independently. That's a painful choice because obviously it reduces open source sharing opportunities. We do open source for a reason, and you can see much evidence that we care about that in the level of our documentation, in our code quality, and in the careful way we've maintained independent backend packages, including listing component package dependencies, providing full browser layer isolation and most recently providing clean uninstallers for all our packages.
Plone Intranet is a huge investment, and we're donating all our code to the Plone community. We hope to establish a strong intranet sub-community while at the same time strengthening the vibrancy of the Plone community as a whole.
The Change Factory is a platform where environmental activists share knowledge
The site offers an online knowledge platform for the entire Dutch environmental movement. The Change Factory is structured using a classical watch-learn-do approach:
- The Knowledge Base provides searchable background information on various topics.
- The Toolbox offers step-by-step help in organising a new initiative.
- The Network connects activists with each other and facilitates learning by sharing practical experiences.
The site is currently only available in Dutch. See the Dutch intro video to get a feel:
Knowledge Management in action
Effective knowledge management and sharing of knowledge results not just from publishing documents ("explicit knowledge"). Much learning and knowledge sharing results from the interactions between people, in which they exchange "tacit knowledge" - stuff you know but didn't know you knew.
The Change Factory is designed to support both aspects, in the form of a knowledge base with documents (explicit knowledge) on the one hand, and a social network geared towards conversation and interpersonal contact (implicit knowledge sharing) on the other hand. A toolbox with learning tools connects both aspects into a learning resource.
As a knowledge platform, the site supports a cycle of knowledge flow and knowledge creation following the well-known SECI model:
- Socialization: sharing knowledge in conversation.
- The network promotes direct contact and dialogue between environmental activists, by not only describing the what of an activity, but also who the organisors are and presenting their contact info. Additionally, a discussion facility on the network makes it easy to exchange experiences.
- Externalisation: writing down your knowledge.
- The network is built around the exchange of "experiences", documenting an action format, so people can learn from successes in another town and replicate the format. This helps to articulate tacit knowledge into explicit knowledge.
- Combination: searching and finding knowledge.
- The searchable knowledge base, organised by theme, facilitates the re-use of knowledge. Documented action formats in the network all follow the same stepwise model, making it easy to mix and match steps from various formats in creating your own activity.
- Internalization: turning learning into action.
- The toolbox with process support documentation helps you assimilate best practices by bringing them into practice, following a simple four-step plan. Here, you absorb explicit know-what knowledge and internalize that into tacit know-how.
De combination of these approaches turns The Change Factory into much more than just a set of documents. The site is a living whole where people communicate and help each other to become more effective, in facilitating the transition of our society to a more sustainable world.
Following the initial project brief, Cosent performed design research in the form of a series of interviews with intended users of The Change Factory. These interviews inquired into the way activists collaborate and communicate in practice, focusing on what people actually need and how a online platform could contribute to their success.
What emerged from the research, is that nobody wants more long documents. Nor was there any need for a marketplace-like exchange of services and support. Rather the interviewees articulated a need for quick-win snippets of actionable knowledge that can immediately be put into practice.
Based on the outcomes of this research, we introduced the Network aspect of the platform: a social network centered on the sharing of succesful action formats, structured in a way that facilitates dialogue, direct contact, and re-use of proven formats across multiple cities.
After articulating this concept into an interaction design and visual design, Cosent built the site in Plone CMS. An initial crew of editors seeded the site with initial content. Recently, the platform was publicly unveiled and immediately attracted scores of active users.
Site: The Change Factory
Concept: Milieudefensie and Cosent.
Interaction design and realisation: Cosent.
Visual design and logo: Stoere Binken Design.
Sprinting in München transformed both the team and the code base of Plone Intranet.
The Plone Intranet project represents a major investment by the companies that together form the Plone Intranet Consortium. Last week we gathered in München and worked really hard to push the Mercury milestone we're working on close to an initial release.
Mercury is a complex project, challenging participants out of their comfort zones in multiple ways:
- Developers from 6 different countries are collaborating remotely, across language barriers and time zones.
- People are collaborating not within their "own" home team but across company boundaries, with people whom they haven't really collaborated with before, and who have not only a different cultural background but also a different "company coding culture" background.
- The backend architecture is unlike any stack peope are used to work with. Instead of "normal" content types, you're dealing with async and BTrees.
- The frontend architecture represents a paradigm shift as well, requiring a significant change in developer attitude and practices. Many developers are used to work from the backend forward; we are turning that on it's head. The design is leading and the development flow is from frontend to backend.
So we have a fragmented team tackling a highly challenging project. The main goal we chose for the sprint therefore, was not to only produce code but more importantly to improve team dynamics and increase development velocity.
Monday we started with getting everybody's development environments updated. Also, Cornelis provided a walkthrough of how our Patternslib-based frontend works. Tuesday the marketing team worked hard on positioning and communications, while the developer teams focused on finishing open work from the previous sprint. As planned, we used the opportunity to practice the Scrum process in the full team to maximize the learning payoff for the way we collaborate. Wednesday we continued with Scrum-driven development. Wednesday afternoon, after the day's retrospective, we had a team-wide disussion resulting in a big decision: to merge all of our packages into a single repository and egg.
The big merge
Mercury consists of 20 different source code packages, each of which had their own version control and their own build/test tooling. This has some very painful downsides:
- As a developer you need to build 20 separate testing environments. That's a lot of infrastructure work, not to mention a fiendishly Jenkins setup.
- When working on a feature, you're either using a different environment than the tests are run in, or you're using the test environment but are then unable to see the integrated frontend results of your work.
- Most user stories need code changes across multiple packages, resulting in multiple pull requests that each depend on the other. Impossible to not break your continuous integration testing that way.
- We had no single environment where you could run every test in every package at once.
So we had a fragmented code base which imposed a lot of infrastructure work overhead, created a lot of confusion and cognitive overhead, actively discouraged adequate testing, and actively encouraged counterproductive "backend-up" developer practices instead of fostering a frontend-focused integrative effort.
Of course throwing everything into a big bucket has it's downsides as well, which is why we discussed this for quite some time before taking our decision.
The main consideration is code re-use and open source community dynamics. Everybody loves to have well-defined, loosely coupled packages that they can mix and match for their own projects. Creating a single "big black box" ploneintranet product would appear to be a big step backward for code re-use.
However, the reality we're facing is that the idea of loosely coupled components is not how the code actually behaves. Sure, our backend is loosely coupled. But the frontend is a single highly integrated layer. We're building an integrated web application, not a set of standalone plugins.
We've maintained the componentized approach as long as we could, and it has cost us. A good example is plonesocial: different packages with well-defined loosely coupled backend storages. But most of our work is in the frontend and requires you to switch between at least 3 packages to make a single frontend change.
In addition, these packages are not really pluggable anymore in the way Plone devs are used to. You need the ploneintranet frontend, you need the ploneintranet application, to be able to deliver on any of it's parts. Keeping something like plonesocial.activitystream availabe as a separately installable Plone plugin is actively harmful in that it sets wrong expectations. It's not independently re-usable as is, so it should not be advertised as such.
We see different strategies Plone integrators can use ploneintranet:
You take the full ploneintranet application and do some cosmetic overrides, like changing the logo and colours of the visual skin.
You design and develop a new application. This starts with a new or heavily customized frontend prototype, which you then also implement the backend for. Technically you either fork and tweak ploneintranet, or you completely build your own application from scratch, re-using the ploneintranet parts you want to keep in the backend via library mode re-use, see below.
Library mode cherry-picking.
You have a different use case but would like to be able to leverage parts of the ploneintranet backend for heavy lifting. Your application has a python dependency on those parts of ploneintranet you want to re-use: via ZCML and GenericSetup you only load the cherries you want to pick.
Please keep in mind, that this situation is exactly the same for the companies who are building ploneintranet. We have those same 3 options. In addition there's a fourth option:
Your client needs features which are not currently in ploneintranet but are actually generally useful good ideas. You hire the ploneintranet designer to design these extensions, and work with the ploneintranet consortium to develop the new features into the backend. You donate this whole effort to ploneintranet; in return you get reduced maintenance cost and the opportunity to re-use the ploneintranet application as a whole without having to do a full customization.
You'll have to join the Plone Intranet Consortium in order to pursue this fourth strategy. But again, there's no difference for current members: we had to join as well.
To make individual component re-use possible, we've maintained the package separation we already had - ploneintranet may be one repository, one egg, but it contains as separate python packages the various functional components: workspace, microblog, document preview, etc. So we do not subscribe to JBOC: Just a Bunch of Code. We don't throw everything into one big bucket but are actively investing in maintaining sane functional packages.
A variant of cherry-picking is, to factor out generically re-usable functionality into a standalone collective product. This will generally only be viable for backend-only, or at least frontend-light functionality, for the reasons discussed above. A good example is collective.workspace: the ploneintranet.workspace implementation is not a fork but an extension of collective.workspace. This connection enables us to implement all ploneintranet specific functionality in ploneintranet.workspace, but factor all general improvements out to the collective. That has already been done and resulted in experimental.securityindexing.
On Thursday we announced a feature freeze on the whole stack, worked hard to get all tests to green and then JC performed the merge of all ploneintranet.* into the new ploneintranet consolidated repo. Meanwhile Guido prepared the rename of plonesocial.* to ploneintranet.*. On Friday we merged plonesocial into ploneintranet and spent the rest of the day in hunting down all test regressions introduced by the merges. Because we now have a single test runner across all packages that meant we also identified and had to fix a number of test isolation problems we hadn't seen before.
Friday 20:45 all tests were finally green on Jenkins!
We still have to update the documentation to reflect the new consolidated situation.
In terms of team building this sprint has been phenomenal. We've been sprinting on ploneintranet for five months now, but this was the first time we were physically co-located and that's really a completely different experience. We already did a lot of pair programming remotely, but it's better if you are sitting next to each other and are actually looking at the same screen. Moreover feeling the vibe in the room is something you cannot replicate remotely. The explosion of energy and excited talking after we decided to do the consolidation merge was awesome.
On top of that we now have a consolidated build, and I can already feel in my own development the ease of mind from knowing that the fully integrated development environment I'm working in is identical to what all my team members are using, and is what Jenkins is testing. Instead of hunting for branches I can see all ongoing work across the whole project by simply listing the ploneintranet branches. Reviewing or rebasing branches is going to be so much more easier.
On top of all that we also made significant progress on difficult features like the document previewing and complex AJAX injections in the social stream.
We started with a fragmented team, working on a fragmented code base. We now have a cohesive team, working on a unified code base. I look forward to demoing Mercury in Sorrento in a few weeks.
- Stay Sharp
Receive updates on innovation, knowledge management and social technology every two months.
- Tag Cloud
- agile complexity continuous integration crowdsourcing design design research enterprise 2.0 foresight innovation internet of things intranet knowledge knowledge management management open source organizational learning plone plonesocial roadmap semantic web social business social network analysis social networking sustainability technology