Vagrant – Upgrading from ubuntu 14.04 to 16.04

Intro

If you don’t have the patience to read to the end, you can find the code that I’m talking about in the post here

The other day I decided that is was time to upgrade my ubuntu 14.04 boxes to 16.04 since LTS end date of support is getting closer ( we are not quite there yet but better to get a head of this ). This should be really strait forward, just update the box string from ubuntu/trusty64 to ubuntu/xenial64 right? Yupp, that easy blog post over… not really though. For some reason canonical decided to change the embedded user from the standard vagrant user with the insecure ssh key to a “ubuntu” user, that can cause problems. First of if you are provisioning these ubuntu images and just want to use your old scripts that rightfully expect the user to be vagrant, they will no longer work. This also caused me problems since my workflow is that I create my own boxes from the official ubuntu box where I provision the software that I want to have on them and after the change the exported boxes no longer work.

Since I already posted creating local baseboxes I’m not going to cover that again, but just the hack around getting rid of the unwanted ubuntu user and adding in the default vagrant user.

The hack

This is by no means an elegant solution, but it works. Then again the most elegant way to handle this is simply for canonical to just have the default vagrant user in there to begin with.

So the actual code is here below, basically what I’m doing is just creating the default vagrant user and deleting the ubuntu user

create-base-box.sh

#!/usr/bin/env bash

BOX_NAME=hx/ubuntu1604

TEST_IF_BOX_EXISTS=$(vagrant box list | grep $BOX_NAME)
if [ "$1" = "-f" ]; then
    TEST_IF_BOX_EXISTS=""
    vagrant box remove $BOX_NAME
    if [ "$?" -ne 0 ]; then
        echo "Could not remove box, check if you have running machines using the box"
        exit 1
    fi
fi

if [ "$TEST_IF_BOX_EXISTS" = "" ]; then
    echo "$BOX_NAME does not exist, creating $BOX_NAME"
    mv Vagrantfile Vagrantfile.tmp
    cp Vagrantfile.BaseBox Vagrantfile
    vagrant up basebox
    vagrant halt basebox
    sed -i 's|#basebox.ssh.|basebox.ssh.|g' Vagrantfile
    vagrant up basebox
    PORT=$(vagrant port | grep '    22 (guest) => ' | awk '{print $5}')
    ssh -o "StrictHostKeyChecking no" vagrant@127.0.0.1 -p $PORT -i keys/vagrant "sudo /vagrant/scripts/remove-user.sh ubuntu"
    vagrant halt basebox
    VBOX_INSTANCE=$(vboxmanage list vms | grep -oP "${PWD##*/}_basebox_[0-9_]*")
    vagrant package --base $VBOX_INSTANCE --output base.box
    # Handle paths differently when using cygwin
    if $(pwd | grep -q cygdrive) ; then
      BOX_FILE_PATH=$(echo "$PWD" | sed -E 's/\/cygdrive\/([a-z])(.*)/file:\/\/\/\1:\2\/base.box/')
    else
      BOX_FILE_PATH=$(pwd | awk '{print "file:" $1 "/base.box"}')
    fi
    vagrant box add $BOX_NAME $BOX_FILE_PATH
    vagrant destroy -f
    rm -f base.box
    rm Vagrantfile
    mv Vagrantfile.tmp Vagrantfile
    echo "Done creating basebox..."
else
    echo "$BOX_NAME exists, moving on..."
fi

These lines here are a bit of a hack

PORT=$(vagrant port | grep '    22 (guest) => ' | awk '{print $5}')
    ssh -o "StrictHostKeyChecking no" vagrant@127.0.0.1 -p $PORT -i keys/vagrant "sudo /vagrant/scripts/remove-user.sh ubuntu"

but I could not get the private_key_path to work, it always prompted me for a password so this is the way I did it.

Then there are two scripts I use, one for creating the vagrant user and one for deleting the ubuntu user.

create-vagrant-user.sh

#!/usr/bin/env bash
echo "Renameing user"
groupadd vagrant
su -c "useradd vagrant -s /bin/bash -m -g vagrant -G vagrant"
chown -R vagrant:vagrant /home/vagrant
echo "vagrant:vagrant" | chpasswd
adduser vagrant sudo
echo %vagrant ALL=NOPASSWD:ALL > /etc/sudoers.d/vagrant
chmod 0440 /etc/sudoers.d/vagrant
usermod -a -G sudo vagrant
mkdir -p /home/vagrant/.ssh
chmod 700 /home/vagrant/.ssh
touch /home/vagrant/.ssh/authorized_keys
chmod 600 /home/vagrant/.ssh/authorized_keys
chown -R vagrant:vagrant /home/vagrant
curl -Ss -L https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub >> /home/vagrant/.ssh/authorized_keys
echo "Done creating user"

remove-user.sh

#!/usr/bin/env bash
echo "Removing user $1"
userdel -r $1
rm -f /etc/sudoers.d/$1
echo "Done removing user"

Now we have to change the Vagrantfile.BaseBox ( see here for details on the pattern used. ). We have to add in two commented out lines that will be used in the create-base-box.sh. When we are provisioning in the software in the first step we log in with the ubuntu user, but when we are deleting that user, we have to log in with the vagrant user ( can’t delete the currently logged in user )

Vagrantfile.BaseBox

Vagrant.configure("2") do |config|
  config.vm.define "basebox" do |basebox|
    basebox.vm.box = "ubuntu/xenial64"
    basebox.vbguest.auto_update = false
    basebox.vm.synced_folder ".", "/vagrant", type: "virtualbox"
    basebox.ssh.insert_key = false
    #basebox.ssh.username = "vagrant"
    #basebox.ssh.password = "vagrant"
    basebox.vm.provision "shell", path: "scripts/create-vagrant-user.sh"
    #And add some provision steps you want here
  end
end

And now when you run the create-base-box.sh you create and add the vagrant box to your vagrant boxes. Then you can use the box like so.

Vagrantfile

Vagrant tricks – overwrite variables

In this post I will go over a useful vagrant trick I have been using, overwriting variables in the Vagrantfile.
Since the vagrant files are essentially just ruby files, we can take advantage of that and overwrite variables quite easily allowing us to tune the infrastructure we are building up. We can define variables in the Vagrantfile with some defaults and have a separate ruby file that we require in if it is present where we overwrite these defaults.

Example

Vagrantfile

$worker_box_name = "centos/7"
$worker_count = 3
$worker_memory = 1024
CLUSTER_CONFIG = File.expand_path("cluster-config.rb")
if File.exist?(CLUSTER_CONFIG)
  require CLUSTER_CONFIG
end
Vagrant.configure(2) do |config|
  (1..$worker_count).each do |i|
    config.vm.define "w#{i}" do |w|
      w.vm.box = $worker_box_name
      w.vm.hostname = "w#{i}"
      w.vm.provider "virtualbox" do |vb|
        vb.memory = $worker_memory
      end
      # provision steps...
    end
  end
end

cluster-config.rb

$worker_count = 2
$worker_memory = 2048

This will create a cluster with 2 workers with 2048mb memory instead of the default 3 worker with 1024mb memory

Vagrant tricks – Creating local baseboxes

In this blog post I am going to go over a useful trick that I have been using to minimize the time of provisioning a vagrant box. Basically what it is about is having two vagrant or more vagrant files in the project (depending on what you are doing) and a script that creates a vagrant box from one of the vagrant files, then use the box in the next one.

Example

We are going to setup a machine with some applications that takes some time to install (can be what ever). Then we are going to configure the installed applications after they are installed, that will be our second vagrant script. Our first vagrant file will be named Vagrantfile.Basebox and our main vagrant file will be named Vagrantfile.

Vagrantfile.Basebox

$centos_box = "centos/7"
Vagrant.configure(2) do |config|
  config.vm.define "basebox" do |basebox|
    basebox.vm.box = $centos_box
    basebox.ssh.insert_key = false
    basebox.vm.synced_folder ".", "/vagrant", type: "virtualbox"
    basebox.vm.provision :shell, path: "long-running-task.sh"
    # more provisioning if needed
  end
end

This will be the script that creates the base box. I chose centos because there is a little quirk that needs to be addressed ( addressed in the comments in the script )

create-basebox.sh

#!/usr/bin/env bash

BOX_NAME=hx/base-centos

TEST_IF_BOX_EXISTS=$(vagrant box list | grep $BOX_NAME)
if [ "$1" = "-f" ]; then
    TEST_IF_BOX_EXISTS=""
    vagrant box remove $BOX_NAME
    if [ "$?" -ne 0 ]; then
        echo "Could not remove box, check if you have running machines using the box"
        exit 1
    fi
fi
if [ "$TEST_IF_BOX_EXISTS" = "" ]; then
    echo "$BOX_NAME does not exist, creating $BOX_NAME"
    mv Vagrantfile Vagrantfile.tmp
    cp Vagrantfile.base.rb Vagrantfile
    vagrant up basebox
    vagrant halt basebox
    # Centos specific, if you are using a different distro,
    # the next couple of lines are not necessary 
    vagrant up basebox
    vagrant halt basebox
    VBOX_INSTANCE=$(vboxmanage list vms | grep -oP "${PWD##*/}_basebox_[0-9_]*")
    vagrant package --base $VBOX_INSTANCE --output base.box
    # Handle paths differently when using cygwin
    if $(pwd | grep -q cygdrive) ; then
      BOX_FILE_PATH=$(echo "$PWD" | sed -E 's/\/cygdrive\/([a-z])(.*)/file:\/\/\/\1:\2\/base.box/')
    else
      BOX_FILE_PATH=$(pwd | awk '{print "file:" $1 "/base.box"}')
    fi
    vagrant box add $BOX_NAME $BOX_FILE_PATH
    vagrant destroy -f
    # Removing the box some times fails because the file handle has not been 
    # released. So put a short sleep before the remove 
    sleep 3
    rm -f base.box
    rm Vagrantfile
    mv Vagrantfile.tmp Vagrantfile
    echo "Done creating $BOX_NAME"
else
    echo "$BOX_NAME exists, moving on..."
fi

Then we can use the base box in the our Vagrantfile.

Vagrantfile

$custom_box = "hx/base-centos"
Vagrant.configure(2) do |config|
  config.vm.define "custombox" do |custombox|
    custombox.vm.box = $custom_box
    custombox.ssh.insert_key = false
    custombox.vm.synced_folder ".", "/vagrant", type: "virtualbox"
    custombox.vm.provision :shell, path: "config-task.sh"
    # more provisioning if needed
  end
end

This way you can do the bulk of the work in the basebox, cutting down on the time that it takes to provision the boxes that build on top of it.

Usage

./create-basebox.sh
vagrant up

BASH browser-like navigation

Note

If you want the fast version just use the finished product without reading the blog post, you can run this command ( at you own risk ) and enable the bash browser like navigation in your shell.
cp ~/.bashrc ~/.bashrc.bak ; curl -k -Ss https://raw.githubusercontent.com/HX-Rd/browser-bash-navigation/master/browserrc >> ~/.bashrc ; source ~/.bashrc

For instructions on how to use the functionality you can head to the git repository
here or scroll down to the bottom of the blog post

In this post I wanted to try to implement a web browser like navigation. What i mean by that is that i want to be able to go back and forward through the directories that i have changed into (like the back and forward buttons in the browser).

Then the functions that I will need to implement is

  • Back
  • Forward
  • Goto ( jump to a certain location in the history )
  • History ( history of previously visited paths )
  • Clear history

The bash shell has some directory stack builtins that can help achieve the set goal. I’m going to do this using these builtins.

What these builtins do is that they push directories to a stack and change into these directories once you push them there. Then you can pop the stack and change into that directory.

The problem with using these builtins out of the box is that you have to change your workflow in order to use them, meaning you cant simply just use cd like you are use to. Now you could just alias the pushd to the cd command but that won’t achieve the goals listed out above. We have to treat the stack more like a list to be able to navigate it like we set out to do. What we need is a stack pointer to be able to know our current location in the stack and then we can traverse through the stack. We will also have keep track on how big the stack is so we can catch if the goto function goes out of bounce. Here below is the code that will achieve the set out goals.

The code

DIR_STACK_POINTER=0
DIR_STACK_SIZE=0
pushd()
{
  if [ $# -eq 0 ]; then
    DIR="${HOME}"
  else
    DIR="$1"
  fi

  builtin pushd "${DIR}" > /dev/null
  DIR_STACK_POINTER=0
  DIR_STACK_SIZE=$(dirs -v | tail -n 1 | awk '{print $1}')
}

pushd_builtin()
{
  builtin pushd > /dev/null
}

popd()
{
  builtin popd > /dev/null
}

stack_list()
{
  dirs -v | awk -v stackp=$DIR_STACK_POINTER '{
    if ($1 == stackp)
      if ($1 == 0)
        printf "%-25s%s\n","Current dir",$2
      else
        printf "%-15s%-10s%s\n","->",$1,$2
    else
      if ($1 == 0)
        printf "%-25s%s\n","Current dir",$2
      else
        printf "%-15s%-10s%s\n","  ",$1,$2
  }'
}

list_stack_reverse()
{
  stack_list | awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }'
}

back_stack()
{
  if [ $(($DIR_STACK_POINTER)) -lt $(($DIR_STACK_SIZE)) ]
  then
    if [ $(($DIR_STACK_POINTER)) -eq 0 ]
    then
      builtin pushd $PWD > /dev/null
      ((DIR_STACK_POINTER++))
      DIR_STACK_SIZE=$(dirs -v | tail -n 1 | awk '{print $1}')
    fi
    ((DIR_STACK_POINTER++))
    NEXT_DIR_POINTER=$( dirs -v | grep -w "$DIR_STACK_POINTER\s\s" | awk ' {print $2}' )
    if [ "$NEXT_DIR_POINTER" = "~" ]
    then
      \cd $HOME
    else
      \cd $NEXT_DIR_POINTER
    fi
  fi
  list_stack_reverse
}

forward_stack()
{
  if [ $(($DIR_STACK_POINTER)) -gt 0 ]
  then
    ((DIR_STACK_POINTER--))
    if [ $(($DIR_STACK_POINTER)) -gt 1 ]
    then 
      NEXT_DIR_POINTER=$( dirs -v | grep -w "$DIR_STACK_POINTER\s\s" | awk ' {print $2}' )
      if [ "$NEXT_DIR_POINTER" = "~" ]
      then
        \cd $HOME
      else
        \cd $NEXT_DIR_POINTER
      fi
    else
      builtin popd > /dev/null
      DIR_STACK_SIZE=$(dirs -v | tail -n 1 | awk '{print $1}')
      DIR_STACK_POINTER=0
    fi
  fi
  list_stack_reverse
}


goto_stack()
{
  DIR_STACK_SIZE=$(dirs -v | tail -n 1 | awk '{print $1}')
  if [ $(($DIR_STACK_SIZE)) -lt $(($1)) ]
  then
    echo "The stack is smaller than argument"
  else
    GOTO_POINTER=$1
    if [ $(($DIR_STACK_POINTER)) -eq 0 ]
    then
      builtin pushd $PWD > /dev/null
      DIR_STACK_SIZE=$(dirs -v | tail -n 1 | awk '{print $1}')
      ((GOTO_POINTER++))
    fi
    if [ $(($GOTO_POINTER)) -gt 1 ]
    then 
      NEXT_DIR_POINTER=$( dirs -v | grep -w "$GOTO_POINTER\s\s" | awk ' {print $2}' )
      if [ "$NEXT_DIR_POINTER" = "~" ]
      then
        \cd $HOME
      else
        \cd $NEXT_DIR_POINTER
      fi
      DIR_STACK_POINTER=$GOTO_POINTER
    else
      builtin popd > /dev/null
      DIR_STACK_SIZE=$(dirs -v | tail -n 1 | awk '{print $1}')
      DIR_STACK_POINTER=0
    fi
  fi
  list_stack_reverse
}

clear_directory_stack()
{
  dirs -c
  DIR_STACK_SIZE=0
  DIR_STACK_POINTER=0
  list_stack_reverse
}

alias cd='pushd'
alias bb='back_stack'
alias ff='forward_stack'
alias ll='list_stack_reverse'
alias gg='goto_stack'
alias dcl='clear_directory_stack'

So the basic thing here to note here is that we have to keep track of the stack size and position where we are in the stack. We also have to translate the ~ to the home folder since we cant use it in the .bashrc file. I started out using tac for reversing the output from dirs -v but to be more posix compliant I switch to this awk expression instead.

awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }'

How to use the functionality.

cd around and get something in the stack.

  • cd -> pushes into the stack
  • bb -> change to a previous directory
  • ff -> change to a directory you where in before you went back with bb
  • ll -> list the stack contents
  • gg (number in stack) -> hop to a indicated place in the stack
  • dcl -> clear the directory stack