Walter, a simple build pipeline tool

Today I announce Walter, a simple build pipeline tool with a collaborator @takahi_i.

Beriefly, Walter is a tiny tool for executing build pipelines (such as automated build workflows) in Continuous Integration / Delivery(CI / CD).

 Background of development

 Build pipeline settings heavily depends on Jenkins GUI

When managing a CI environment with Jenkins, we commonly use some sort of build pipeline plugins (ex. buildpipeline-plugin) to define deployment flows and execute the tasks in the flow one by one.

Although these Jenkins plugins are useful to manage and control CI, we still have some problems in the deployment of our services.

Although we have built our system that maintains the servers settings with configuration files of Chef or Ansible in the Git repositories, only the deployment pipeline settings are still in Jenkins server… Modern CI services, [travis-ci][travis], wercker provide the solution to this problem. Unfortunately, we can’t execute these services locally.

To manage the deployment settings with the local repositories, we have developed a tiny deployment workflow engine in Golang.

 About Walter

The followings are the benefits of adopting Walter.

We can adopt Walter without the special training. The pipeline configuration is yaml formatted (It is similar to the configuration file of TravisCI). We just add multiple stages containing shell commands in the pipeline setting.

 Use case

In the settings yaml for Walter, we describe the workflow control and tasks in pipeline: section.
The definition of each task is written in stage: section.

stage: section has the properties below:

The following example shows a sample setting of Walter pipeline:

# Hipchat notification settings
messenger: 
   type: hipchat
   room_id: ROOM_ID
   token: TOKEN
   from: USER_NAME

# Workflow configuration 
pipeline:
    # step 1
    - stage_name: Notify the beggining of process pipeline
      command: echo "build pipeline is started"
      message: true
      run_after:
          # step 2
          -  stage_name: Provisioning server with Ansible.
             command: Ansible-playbook -i hosts site.yml
          -  stage_name: Test application with rspec 
             command: bundle exec rake test
    # step 3
    - stage_name: Deploy application with Capistrano
      command: bundle exec cap production deploy
    # step 4
    - stage_name: Test server settings by serverspec
      command: bundle exec rake spec
    # step 5
    - stage_name: Notify the finish of process pipeline
      message: true
      command: echo "build pipeline is finished"

Walter executes each stage described at the top level of pipiline sequencially.
However there is one exception run_after. Only stages in run_after section are executed in parallel.

The follwoing diagram shows how the some stages in the pipeline execute in palallel.

We also provide one relatively large build pipeline example for Spring demo application deployment.

You can download and run the Walter to understand the usage. Please check it out!

 Try Walter

We provide Walter compiled binaries at github-releases. You can try Walter quickly.

As a demonstration, we show how to execute Walter in Mac.

  1. Get the binary for Mac.

    wget https://github.com/Walter-cd/Walter/releases/download/v0.1.0/walter_darwin_amd64.zip
    unzip walter_darwin_amd64.zip
    
  2. Get the sample of build pipeline settings

    cd walter_darwin_amd64
    wget https://gist.githubusercontent.com/ainoya/78971d4b1e2797a59af6/raw/992467e22597e7a6de5834e6a21bf6c624286970/pipeline_sample.yml
    
  3. Run Walter. you can specify the build pipeline settings file by -c option. (by default, use ./pipeline.yml)

    ./walter -c pipeine_sample.yml
    

When you run Walter and execute the build pipelines and find the bugs, please submit to GitHub issues :)

 Microservices with Walter

If a system architecture consists of multiple services / servers, it has often deploy dependencies each other.

In such cases, Walter helps us to deploy the services with multiple components. We can deploy the services following deploy dependencies automatically with Walter as the below diagram.

Actually, Ansible already has wait_for of parallel execution option for clustering multiple services. However Ansible playbooks become huge and intractable in our existing services. Therefore we think it is a good idea to split huge playbooks with Walter to make build pipelines more maintainable.

 Future plans

Though The Code quality of Walter is still prototype level, we’re going to improve the quality periodically. Especially we will enhance error handring and retrying feature.

Furthermore, we will provides below features:

We hope you to try Walter, and contribute the development of Walter. Let’s improve build pipeline system with us!

Thanks.

 
252
Kudos
 
252
Kudos

Now read this

Access Docker API from within container

Nowadays I’m contributing to pool project, which provides the simple http proxy launches docker container on access with commit-id of git repository, to be called even a “Just In Time Container”. At first prototyping, pool provided the... Continue →