Walter, a simple build pipeline tool
Today I announce Walter, a simple build pipeline tool with a collaborator @takahi_i.
Beriefly, Walter is a tiny tool for executing build pipelines (such as automated build workflows) in Continuous Integration / Delivery(CI / CD).
Background of development
Build pipeline settings heavily depends on Jenkins GUI
When managing a CI environment with Jenkins, we commonly use some sort of build pipeline plugins (ex. buildpipeline-plugin) to define deployment flows and execute the tasks in the flow one by one.
Although these Jenkins plugins are useful to manage and control CI, we still have some problems in the deployment of our services.
- None of us is able to run build pipeline without our Jenkins server
- No one is able to check if the pipeline works until starting the CI jobs in Jenkins server
- We feel the difficulties to migrate the build pipeline settings to other CI environments
Although we have built our system that maintains the servers settings with configuration files of Chef or Ansible in the Git repositories, only the deployment pipeline settings are still in Jenkins server… Modern CI services, [travis-ci][travis], wercker provide the solution to this problem. Unfortunately, we can’t execute these services locally.
To manage the deployment settings with the local repositories, we have developed a tiny deployment workflow engine in Golang.
The followings are the benefits of adopting Walter.
- Store build pipeline as a settings file: Walter makes the deployment easy, more portable in managing CI settings.
- Execute build pipeline locally: We can run CI workflows without a CI server.
- Simple to use: Does anyone like to understand horribly complex GUIs or DSLs? With Walter, we just need shell and yaml.
We can adopt Walter without the special training. The pipeline configuration is yaml formatted (It is similar to the configuration file of TravisCI). We just add multiple stages containing shell commands in the pipeline setting.
In the settings yaml for Walter, we describe the workflow control and tasks in
The definition of each task is written in
stage: section has the properties below:
stage_name: Name of stage.
stage_type: Type of stage.
commandis applied by default when
stage_typeis not specified.
command: Description of the system command to execute.
run_after: Definition of the stages that you want to execute after finished executing parent
run_afterhas more than one
stagesections. the stages in
run_aftersection is executed in parallel.
The following example shows a sample setting of Walter pipeline:
# Hipchat notification settings messenger: type: hipchat room_id: ROOM_ID token: TOKEN from: USER_NAME # Workflow configuration pipeline: # step 1 - stage_name: Notify the beggining of process pipeline command: echo "build pipeline is started" message: true run_after: # step 2 - stage_name: Provisioning server with Ansible. command: Ansible-playbook -i hosts site.yml - stage_name: Test application with rspec command: bundle exec rake test # step 3 - stage_name: Deploy application with Capistrano command: bundle exec cap production deploy # step 4 - stage_name: Test server settings by serverspec command: bundle exec rake spec # step 5 - stage_name: Notify the finish of process pipeline message: true command: echo "build pipeline is finished"
Walter executes each stage described at the top level of
However there is one exception
run_after. Only stages in
run_after section are executed in parallel.
The follwoing diagram shows how the some stages in the pipeline execute in palallel.
We also provide one relatively large build pipeline example for Spring demo application deployment.
You can download and run the Walter to understand the usage. Please check it out!
We provide Walter compiled binaries at github-releases. You can try Walter quickly.
As a demonstration, we show how to execute Walter in Mac.
Get the binary for Mac.
wget https://github.com/Walter-cd/Walter/releases/download/v0.1.0/walter_darwin_amd64.zip unzip walter_darwin_amd64.zip
Get the sample of build pipeline settings
cd walter_darwin_amd64 wget https://gist.githubusercontent.com/ainoya/78971d4b1e2797a59af6/raw/992467e22597e7a6de5834e6a21bf6c624286970/pipeline_sample.yml
Run Walter. you can specify the build pipeline settings file by
-coption. (by default, use
./walter -c pipeine_sample.yml
When you run Walter and execute the build pipelines and find the bugs, please submit to GitHub issues :)
Microservices with Walter
If a system architecture consists of multiple services / servers, it has often deploy dependencies each other.
In such cases, Walter helps us to deploy the services with multiple components. We can deploy the services following deploy dependencies automatically with Walter as the below diagram.
Actually, Ansible already has
wait_for of parallel execution option for clustering multiple services. However Ansible playbooks become huge and intractable in our existing services. Therefore we think it is a good idea to split huge playbooks with Walter to make build pipelines more maintainable.
Though The Code quality of Walter is still prototype level, we’re going to improve the quality periodically. Especially we will enhance error handring and retrying feature.
Furthermore, we will provides below features:
- Communicate data between jobs: We will try message-passing feature between jobs with goroutine
- More simple definition of settings: Settings by YAML is still complex; we improve the the syntax to make it easy.
- Ready to ChatOps: We will enhance notification to various sevices (such as slack) and more details of notification settings.
- Integration with Jenkins: Feedback mechanism to tell the progress of executing Walter build pipeline to Jenkins (or other CI service) is provided.
We hope you to try Walter, and contribute the development of Walter. Let’s improve build pipeline system with us!