Tebako: an image packager

Purpose

Tebako is an executable packager. It packages a set of files into a DwarFS file system for read-only purposes.

After packaging the file system into an image, Tebako produces a single executable binary that allows the user to execute a selected file from the packaged software from a point in the file system.

The packaged binary should support:

Packaging a default DwarFS image inside the binary
Support signing of the binary on macOS (via notarization)

In the future:

Downloading new DwarFS images to be stored in the local home directory
Allowing loading multiple DwarFS images in a stacked way
Supporting a COW mechanism that the newly written files are stored in a separate image that can be loaded on top of the read-only file systems.

Supported platforms

The Tebako packager is tested on the following platforms:

Linux: Ubuntu 20.04; Alpine 3.17
MacOS: macOS 12 (Monterey), 13 (Ventura)
Windows: 2019, 2022 (using MinGW ucrt64 toolchain)

Please note that Windows support is in pre-release and is a subject of further testing and optimization

Supported Ruby versions

The Tebako packager supports the following versions of Ruby for packaging:

2.7.8 (Linux, MacOS)
3.0.6 (Linux, MacOS)
3.1.4 (Linux, MacOS, Windows)
3.2.3 (Linux, MacOS, Windows)

Support of specific version including minor release requires some effort, sometimes extensive but our goal is to be able to package all maintained Ruby releases.

Prerequisites

Ubuntu 20.04

GNU C/C 10+ or Clang C/C 12+

apt install -y gcc-10 g++-10
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10

apt install -y clang-12
update-alternatives --install /usr/bin/clang clang /usr/bin/clang-12 150
update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-12 150

CMake version 3.20+

Tebako relies on CMake 3.20+, which may not be available as a default package.

If it is not available as default package it can be set up as follows:

apt-get remove --purge --auto-remove cmake
apt-get update
apt-get install -y software-properties-common lsb-release curl
apt-get clean all
curl https://apt.kitware.com/kitware-archive.sh | bash
apt-get install cmake

Other development tools and libraries

apt-get -y install sudo git curl build-essential pkg-config bison flex autoconf  \
   binutils-dev libevent-dev acl-dev libfmt-dev libjemalloc-dev libiberty-dev    \
   libdouble-conversion-dev liblz4-dev liblzma-dev libssl-dev libunwind-dev      \
   libboost-filesystem-dev libboost-program-options-dev libboost-system-dev      \
   libboost-iostreams-dev  libboost-date-time-dev libboost-context-dev           \
   libboost-regex-dev libboost-thread-dev libbrotli-dev libdwarf-dev libelf-dev  \
   libgoogle-glog-dev libffi-dev libgdbm-dev libyaml-dev libncurses-dev          \
   libreadline-dev libncurses-dev libreadline-dev ruby-dev ruby-bundler          \
   libutfcpp-dev

Alpine 3.17

apk --no-cache --upgrade add build-base cmake git bash autoconf boost-static   \
   boost-dev flex-dev bison make binutils-dev libevent-dev acl-dev sed python3 \
   pkgconfig lz4-dev openssl-dev zlib-dev xz ninja zip unzip curl libdwarf-dev \
   libunwind-dev gflags-dev elfutils-dev libevent-static openssl-libs-static   \
   lz4-static xz-dev zlib-static libunwind-static acl-static tar libffi-dev    \
   gdbm-dev yaml-dev yaml-static ncurses-dev ncurses-static readline-dev       \
   readline-static p7zip ruby-dev gcompat gettext-dev gperf brotli-dev         \
   brotli-static jemalloc-dev fmt-dev xz-static

macOS 12 (Monterey)

brew update
brew install gnu-sed bash pkg-config bison flex binutils libffi gdbm zlib ncurses \
  double-conversion boost jemalloc fmt glog libevent libsodium lz4 xz libyaml openssl@3

Windows (2019, 2022)

The simplest approach is to use Ruby development environment provided by RubyInstaller, for example Ruby+Devkit 3.1.4-1.

Once it is installed use the following commands:

  ridk enable ucrt64
  pacman -S git tar bison flex toolchain make cmake
            boost diffutils libevent double-conversion
            fmt glog dlfcn gtest autotools ncurses libyaml

Installation

General

Tebako is distributed as a Ruby gem

gem install tebako

Quick setup on Ubuntu 20.04 on Docker

Launch a container on the target platform:

# For x86_64
docker run -it --platform linux/x86_64 ubuntu bash

# For Apple M1
docker run -it --platform linux/aarch64 ubuntu bash

In the container:

export DEBIAN_FRONTEND=noninteractive
export TZ=Etc/UTC

apt-get update
apt-get install -y software-properties-common
add-apt-repository -y ppa:ubuntu-toolchain-r/test
apt-get install -y gcc-10 g++-10

apt-get install -y curl git ruby ruby-dev pkg-config bison flex make autoconf
curl https://apt.kitware.com/kitware-archive.sh | bash
apt-get install -y cmake

apt-get -y install sudo git curl build-essential pkg-config bison flex autoconf \
   binutils-dev libevent-dev acl-dev libfmt-dev libjemalloc-dev libiberty-dev    \
   libdouble-conversion-dev liblz4-dev liblzma-dev libssl-dev libunwind-dev      \
   libboost-filesystem-dev libboost-program-options-dev libboost-system-dev      \
   libboost-iostreams-dev  libboost-date-time-dev libboost-context-dev           \
   libboost-regex-dev libboost-thread-dev libbrotli-dev libdwarf-dev libelf-dev  \
   libgoogle-glog-dev libffi-dev libgdbm-dev libyaml-dev libncurses-dev          \
   libreadline-dev libutfcpp-dev libncurses-dev libreadline-dev gcc-10 g++-10    \
   ruby-dev ruby-bundler

gem install tebako

Usage

Commands

Installation

gem install tebako

Press

This command "presses" a Ruby project using the Tebako setup from the Tebako root folder (<tebako-root-folder>). Please note that upon the first invocation of press command tebako collects required packages, builds the and creates packaging environment. This is a lengthly task that can take significant time, up to 1 hour. Upon the next invocation tebako will use previously created packaging environment. The press process itself takes minutes. You can manage setup of packaging environment manually; please refer to description of setup and clean cmmands below.

tebako press     \
  [-p|--prefix=<tebako-root-folder>] \
  [-R|--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>]   \
  -r|--root=<project-root-folder>     \
  -e|--entry-point=<entry-point>      \
  [-o|--output=<packaged file name>] \
  [-l|--log-level=<error|warn|debug|trace>]

Where:

<tebako-root-folder>, the Tebako setup folder (optional, defaults to current folder)
Ruby parameter defines Ruby version that will be packaged (optional, defaults to 3.1.4)
<project-root>, a folder at the host source file system where project files are located
<entry-point>, an executable file (binary executable or script) that shall be started when packaged file is called
output, the output file name (optional, defaults to <current folder>/<entry point base name)
log-level, the logging level for tebako built-in memory filesystem driver (optional, defaults to error)

tebako press \
  --root='~/projects/myproject' \
  --entry=start.rb \
  --output=/temp/myproject.tebako

Setup

Collects required packages, builds the and creates packaging environment. This is a lengthly task that can take significant time, up to 1 hour. Tebako supports several configurations at a single system given that their root directories differ and nultiple Ruby versions within single configuration

This command is optional, tebako creates packaging environment automatically upon the first invocation of press command. However, if you plan to use tebako in CI/CD environment with caching it is highly recommended to build cache based on tebako setup output. Building cache based on tebako press may create inconsistent environment upon restore.

tebako setup     \
  [-p |--prefix=<tebako-root-folder>] \
  [-R |--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>]

Where:

<tebako-root-folder>, the Tebako setup folder (optional, defaults to current folder)
Ruby parameter defines Ruby version that will be packaged (optional, defaults to 3.1.4)

Clean

This command deletes tebako artifacts created by setup and press commands. Normally you do not need to do it since tebako packager optimizes artifacts lifecycle on its own.

tebako clean \
  [-p|--prefix=<tebako-root-folder>]

Where:

<tebako-root-folder>, the Tebako setup folder (optional, defaults to current folder)

tebako clean --prefix='~/.tebako'

Clean ruby

This command deletes tebako Ruby artifacts created by setup and press commands. Dwarfs libraries are not cleaned. Normally you do not need to do it since tebako packager optimizes artifacts lifecycle on its own.

tebako clean_ruby
  [-p|--prefix=<tebako-root-folder>]
  [-R|--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>]

Where:

<tebako-root-folder>, the Tebako setup folder (optional, defaults to current folder)
Ruby parameter defines Ruby version that will cleaned (optional, cleans all versions by default)

tebako clean_ruby --prefix='~/.tebako'

Build script hash

Hash command will calculate tebako script hash that may be used as a cache key in CI/CD environment like GitHub Actions

tebako hash

Exit codes

Code	Condition
0	No error
1	Invalid command line
101	`tebako setup` failed at configuration step
102	`tebako setup` failed at build step
103	`tebako press` failed at configuration step
104	`tebako press` failed at build step
253	Unsupported Ruby version
254	Unsupported operating systems
255	Internal error

Image extraction

Tebako provides an option to an extract filesystem from a package to local folder for verification or execution.

<tebako-packaged-executable> --tebako-extract [<root folder for extracted filesystem>]

Where,

<root folder for extracted filesystem> is optional and defaults to source_filesystem

Extracting Tebako content from the metanorma package:

metanorma --tebako-extract temp-image

The --tebako-extract option actually runs the following Ruby script:

require 'fileutils'
FileUtils.copy_entry '<in-memory filesystem root>', ARGV[2] || 'source_filesystem'

Ruby packaging specification

This is high-level description of the Tebako Ruby packaging mechanism. This specification was inspired by the ruby-packer approach.

Note	For various reasons, Tebako Ruby is a fully separate implementation, no line of code was copied from `ruby-packer`.

Depending on the configuration files that are present in the root project folder, the Tebako Ruby packager support five different scenarios:

Scenario	`*.gemspec`	`Gemfile`	`*.gem`
1	No	No	No
2	No	No	One
3	One	No	Any
4	One	One	Any
5	No	One	Any
Error	No	No	Two or more
Error	Two or more	Any	Any

These scenarios differ in what files are packaged and where the entry point is located, as follows:

Scenario	Description	Packaging	Entry point
1	Simple ruby script	Copy `<project-root>` with all sub-folders to packaged filesystem	`<mount_point>/local/<entry_point base name>`
2	Packaged gem	Install the gem with `gem install` to packaged filesystem	`<mount_point>/bin/<entry_point base name>` (i.e., binstub is expected)
3	Gem source, no `bundler`	Build the gem using `gem build` command at the host Install it with `gem install` to packaged filesystem	`<mount_point>/bin/<entry_point base name>` (i.e., binstub is expected)
4	Gem source, `bundler`	Collect dependencies at the host with `bundle install` Build the gem using `gem build` command Install it with `gem install` to packaged file system	`<mount_point>/bin/<entry_point base name>` (i.e., binstub is expected)
5	Rails project	Deploy project to packaged filesystem using `bundle install`	`<mount_point>/local/<entry_point base name>`

Trivia: origin of name

"tamatebako" (玉手箱) is the treasure box given to Urashima Taro in the Ryugu, for which he was asked not to open if he wished to return. He opened the box upon the shock from his return that three hundred years has passed. Apparently what was stored in the box was his age.

This packager was made to store Ruby and its gems, and therefore named after the said treasure box (storing gems inside a treasure box).

Since "tamatebako" is rather long for the non-Japanese speaker, we use "tebako" (手箱, also "tehako") instead, the generic term for a personal box.

tebako

Runtime

Tebako: an image packager

Purpose

Supported platforms

Supported Ruby versions

Prerequisites

Ubuntu 20.04

GNU C/C 10+ or Clang C/C 12+

CMake version 3.20+

Other development tools and libraries

Alpine 3.17

macOS 12 (Monterey)

Windows (2019, 2022)

Installation

General

Quick setup on Ubuntu 20.04 on Docker

Usage

Commands

Installation

Press

Setup

Clean

Clean ruby

Build script hash

Exit codes

Image extraction

Ruby packaging specification

Trivia: origin of name