Usage and examples

Interactions with the redpesk infrastructure

Project identification in the redpesk infrastructure

In the redpesk infrastructure, a project can be identified by three different fields: ID, name or slug.

  • The ID of the project is an UUID generated by the redpesk infrastructure when the project is created. It is unique.
  • The name of the project is an identifier given by the user to a project. It is not unique and can contains special characters.
  • The slug of the project is either specified by the user or created by the redpesk infrastructure from the project name. It is not unique but cannot contain special characters.

To create the slug, the infrastructure processes the name given by the user. It gets rid of all special characters that are replaced by a dash. And the upper case letters are set to lower case ones.

# List the projects available for the user on redpesk infrastructure
> rp-cli projects list
ID                                     Slug                        Name                        Version        Type
b1a06375-0d28-4091-b9d6-4603cebe85dd   redpesk-ci                  Redpesk CI                  [Not set]      standard
77a6df2a-918a-4fff-a108-8c5f0ab4dc80   redpesk-industrial          Redpesk Industrial          [Not set]      standard
74c88f7a-3f5a-4072-bb29-2291fa8bc05a   redpesk-industrial-image    Redpesk Industrial Image    [Not set]      standard

While UUIDs are great to handle by bots, they are quite challenging for humans. This is why rp-cli handles both the ID and the slug when it comes to identifying projects on the command line.

# Get a project from its slug
> rp-cli projects get redpesk-industrial -v
ID:                      77a6df2a-918a-4fff-a108-8c5f0ab4dc80
Name:                    Redpesk Industrial
Slug:                    redpesk-industrial
Version:                 [Not set]
Type:                    standard
Supported Objects:       applications
Description:             It contains all the industrial applications
Distributions:           redpesk-lts-arz-1.0 [Mand]
Architectures:           x86_64 [Mand], aarch64 [Opt]
Creation time:           2022-04-21 10:30:41.362243 +0000 UTC
Update time:             2022-04-21 10:30:41.362243 +0000 UTC
Bookmarked:              false
Short release naming:    false
Emulated build:          false

# Get a project from its ID
> rp-cli projects get 77a6df2a-918a-4fff-a108-8c5f0ab4dc80 -v
ID:                      77a6df2a-918a-4fff-a108-8c5f0ab4dc80
Name:                    Redpesk Industrial
Slug:                    redpesk-industrial
Version:                 [Not set]
Type:                    standard
Supported Objects:       applications
Description:             It contains all the industrial applications
Distributions:           redpesk-lts-arz-1.0 [Mand]
Architectures:           x86_64 [Mand], aarch64 [Opt]
Creation time:           2022-04-21 10:30:41.362243 +0000 UTC
Update time:             2022-04-21 10:30:41.362243 +0000 UTC
Bookmarked:              false
Short release naming:    false
Emulated build:          false

N.B.: rp-cli can request the user to explicitly identify a project with its UUID if two projects have the same slug on the redpesk infrastructure.

Application identification in the redpesk infrastructure

Application identification on the redpesk infrastructure works in the same way as for projects. They have an ID, a slug and a name.

Additionally, applications have another field named package-name. This field is important because it is the identifier used to install the built package on targets (with dnf install command). The package name needs to be provided by the user during application creation, and needs to be unique within the project.

# Get the application named helloworld-binding
> rp-cli applications get helloworld-binding -v
ID:                   e9cb3f5f-271a-4ce2-b410-5f689ea68ab2
Name:                 Helloworld binding
Package Name:         helloworld-binding
Slug:                 helloworld-binding
Type:                 standard
Parent project:       redpesk-industrial
Description:          [Not set]
Source URL:           https://github.com/redpesk-samples/helloworld-binding.git
Source Revision:      master
External Specfile:    [Not set]
Distributions:        redpesk-lts-arz-1.0 [Mand]
Autotests:            OFF
Autoaudits:           OFF
Audits settings:      {"audit":null}
Creation time:        2022-04-21 10:30:46.496649 +0000 UTC
Update time:          2022-04-21 10:31:02.008922 +0000 UTC
Bookmarked:           false
Emulated build:       false
Services:

 - Name: download_sources
 - Parameters:
    download:      false
    force:         false

 - Name: excludearch
 - Parameters:
    arches:     []

 - Name: import_gitpkg
 - Parameters:
    ref: ""
    url: ""

 - Name: setverrel
 - Parameters:
    release:     true
    version:     true

Handling identifiers in scripts

In order to make scripting easy with rp-cli, it is possible to specify a slug or alias at project or application creation. It allows the script to interact with the created project or application using this slug, as it can be seen in the following example.

# Create a new project with the "my-new-project" alias
> rp-cli projects add -n "New project" -a my-new-project -d "This is a new project" --mandatory-distro redpesk-lts-arz-1.0 --optional-arch x86_64 --mandatory-arch aarch64
-- Project creation requested by user --
Checking that at least one architecture is set...   [OK]
Checking that at least one distribution is set...   [OK]
Creation of "New project" project...                [OK]

# Creation of an application "hello-iot-bzh" within the "my-new-project" project
> rp-cli applications add -n "Hello Iot.bzh" -a hello-iot-bzh --pkg-name helloworld-binding --project my-new-project --source-url https://github.com/redpesk-samples/helloworld-binding.git --specfile-sources conf.d/packaging/helloworld-binding.spec

-- Application creation requested by user --
Processing the distribution(s) settings...          [OK]
Creation of "Hello Iot.bzh" application...          [OK]
Wait for creation confirmation from the backend...  [OK]

# Start the build of the "hello-iot-bzh" application
> rp-cli applications build hello-iot-bzh -v
-- Application build requested by user --
Building the "hello-iot-bzh" application...

[..]

# Start the tests of the "hello-iot-bzh" application
> rp-cli applications test hello-iot-bzh -v
-- Application test requested by user --
Testing the "hello-iot-bzh" application on build 1012...

[..]

Other command examples

Here below one can find a few basic examples of rp-cli use, interacting with the redpesk infrastructure.

# Get the list of all the available architectures in the redpesk instance
rp-cli misc arch

# Get the list of all the available distributions in the redpesk instance
rp-cli misc distributions

# Get the list of all the available projects in the redpesk instance (verbose option)
rp-cli projects list -v

# Get the list of all the available applications in the redpesk instance (non-verbose option)
rp-cli applications list

# Creation of a new project called "IOT Project", without waiting for the acknowledgement events
# Note: the project slug, being based on the project name, will end up being "iot-project"
rp-cli projects add --nonblocking -n "IOT Project" --mandatory-arch x86_64 --optional-arch aarch64 --mandatory-distro redpesk-lts-arz-1.0

# Creation of a new application named "Helloworld binding" within the project whose slug is "iot-project"
rp-cli applications add --name "Helloworld binding" -d "This is my beautiful app" --pkg-name "helloworld-binding" --source-rev "master" --source-url "https://github.com/redpesk-samples/helloworld-binding.git" -p "iot-project"

# Upload a specfile for the application with the slug "helloworld-binding"
rp-cli applications upload helloworld-binding --file-path /home/me/iot-app.spec

# Get the list of all the available applications contained in the project whose slug is "iot-project"
rp-cli applications list -p iot-project

# Start a new build for the application "helloworld-binding", verbose option
rp-cli applications build helloworld-binding -v

# Start a new test for the application "helloworld-binding", on the latest build, verbose option
rp-cli applications test helloworld-binding -v

# Retrieve the builds list for the application "helloworld-binding"
rp-cli applications builds list helloworld-binding

# Start a new test on the build #1068 (buildsID can be seen with 'rp-cli applications builds list')
rp-cli applications test 1068

# Get the list of all the tests for the application "helloworld-binding"
rp-cli applications tests list helloworld-binding

# Get the list of public teams in the redpesk instance, with verbose option
rp-cli teams list --verbose

# Download the logs of the latest build of the application "helloworld-binding" in the directory "/tmp"
rp-cli applications builds logs helloworld-binding -p iot-project -d /tmp

# Download the logs of the latest test of the application "helloworld-binding",
# from the project "my-project" in the current directory
rp-cli applications tests logs helloworld-binding -p iot-project --directory /tmp/

Interactions with the local builder

As it has been said in the introduction, rp-cli now allows to package locally your application before doing it on the redpesk infrastructure. Therefore, it permits to have a quick iteration cycle time while packaging. It also allows to build locally your own custom redpesk image.

In order to do that, the local builder needs to be installed in your host. Please refer to the local builder’s installation chapter for more information about it.

Moreover, rp-cli needs to have the correct local builder’s parameters set. If the configuration have not been done yet, please refer to the rp-cli local builder configuration.

Packaging initialization

Before packaging an application, the user needs to specify some parameters to rp-cli. If the application already exists in the redpesk infrastructure, less parameters are needed. However, rp-cli always needs that at least the project, that will contain the application, exists.

Several examples of initialization can be found here below, corresponding to different cases.

# Initialization for an application already existing in the redpesk infrastructure
> rp-cli local package init redtest-helloworld-api

-- Package initialization requested by user --

Checking application details...		[OK]
Checking packaging directories...	[OK]

"gitsources/" directory used:    /home/devel/sandbox/gitsources
"gitpkgs/" directory used:       /home/devel/sandbox/gitpkgs

Retrieving package configuration for "redtest-helloworld-api"	[OK]
Setting up the local-builder...
Cloning the package sources...

CLONE LOGS >>>
Cloning into 'redtest-helloworld-api'...
remote: Enumerating objects: 80, done.        
remote: Counting objects: 100% (6/6), done.        
remote: Compressing objects: 100% (6/6), done.        
remote: Total 80 (delta 1), reused 0 (delta 0), pack-reused 74        
Receiving objects: 100% (80/80), 18.05 KiB | 3.01 MiB/s, done.
Resolving deltas: 100% (18/18), done.
<<< CLONE LOGS

Working out the spec file...	[OK]
Creation of the sources archives inside local builder...

ARCHIVE LOGS >>>
Archive redtest-helloworld-api-1.0.0.tar.gz not found within /home/devel/gitpkgs/redtest-helloworld-api, force generation...
Will try to checkout to master (commit_checkout)
Fetching repo from /home/devel/gitsources/redtest-helloworld-api
Commit checkout master is set as a RefType.BRANCH ref
Successful checkout to master
Current commit is e751a75a160b49c291c94ced38c4762640630ca7
Version is 1.0.0
Making archive ...
redtest-helloworld-api-1.0.0.tar.gz
<<< ARCHIVE LOGS

The spec file 'redtest-helloworld-api.spec' has been copied to '/home/devel/sandbox/gitpkgs/redtest-helloworld-api'
Please modify '/home/devel/sandbox/gitpkgs/redtest-helloworld-api/redtest-helloworld-api.spec' while working on packaging, only this specfile is used when running 'rp-cli local package run'

# Initialization for an application that does not exist in the redpesk infrastructure
# In this case, the sources are downloaded from the given GIT repository with the given revision
# The spec file is located in the git repository under 'conf.d/packaging/rpm/redtest-helloworld-api.spec'
> rp-cli local package init redtest-helloworld-api --project best-project --source-url http://git.ovh.iot/redpesk/redpesk-samples/redtest-helloworld-api.git --repo-rev master-next --spec-path conf.d/packaging/rpm/redtest-helloworld-api.spec

# Initialization for an application that does not exist in the redpesk infrastructure
# In this case, the package sources (archive + specfile) are downloaded from the given GIT repository on the default rev (master)
> rp-cli local package init helloworld-binding --project best-project --package-url http://iotbzh-git-dev01.lorient.iot/iotbzh-runtime-dev01.lorient.iot/redpesk-samples/helloworld-binding.git

Packaging runs

Once the initialization is correctly done, the application can be packaged with rp-cli. A target needs to be specified (x86_64 or aarch64), the default value being aarch64. The project slug needs to be provided as well.

Several other flags allow to debug interactively the application packaging. For example, the use of --no-clean-after followed, on the next iteration, by the use of --shell allows to enter inside the mock used to build your package and to discover the build environment thanks to a shell. Other useful flags are available for debugging. To learn more, simply run rp-cli local package run -h in your terminal!

Here below, several examples of rp-cli local package run commands.

# Request to recreate the archive and then package the 'redtest-helloworld-api' for x86_64 architecture
> rp-cli local package run redtest-helloworld-api --target x86_64 --regen-archive

# Request to package the 'redtest-helloworld-api' application without cleaning the mock after (allows to debug potential problems)
# The package will be done for aarch64 architecture
> rp-cli local package run redtest-helloworld-api --no-clean-after

Building images locally

The combination of rp-cli and the local builder allows the user to build custom redpesk images. In order to build an image locally, the user first needs to create a parent project. Inside this project an image object can be created, thanks to the rp-cli images add command.

# Creation of the parent project, supporting image type
$ rp-cli projects add -n "Redpesk Industrial Image" -a "redpesk-industrial-image" --mandatory-distro redpesk-lts-arz-1.0 --mandatory-arch x86_64 --optional-arch aarch64 --images
-- Project creation requested by user --
Checking that at least one architecture is set...	[OK]
Checking that at least one distribution is set...	[OK]
Creation of "Redpesk Industrial Image" project...	[OK]
$
$ rp-cli images add --help
Create a new image

Usage:
  rp-cli images add [flags]

Aliases:
  add, a

Flags:
  -a, --alias string          [Optional] Alias for the image name (must be unique)
      --arch string           [Mandatory] Target architecture
  -d, --distribution string   [Mandatory] Distribution "slug"
  -u, --download-url string   [Optional] URL where the image can be downloaded
  -f, --force                 [Optional] Do not run the usual check and create the image anyway
  -h, --help                  help for add
  -k, --kickstart string      [Optional] Kickstart filename, needed to build the image
      --ksurl string          [Optional] URL to the repository containing the kickstart file, needed to build the image
  -m, --model-board string    [Optional] Compatible board model for this image
  -n, --name string           [Mandatory] Name of the image to create
  -p, --project string        [Mandatory] ID of the project linked to the image, the project repo is added at the image build.
  -r, --repo strings          [Optional] Additional rpms repository to add in image, this flag can be used several times
  -s, --security string       [Optional] Security model of the image
  -v, --version string        [Optional] Image version (default: distribution version)

Global Flags:
  -c, --config string        Config file containing the user options (default are $HOME/.redpesk/rp-cli/rp-cli-config.json or /etc/redpesk/rp-cli/rp-cli-config.json)
      --logfilename string   Filename where logs will be redirected (default "stderr")
  -l, --loglevel string      Logging level (supported levels: panic, fatal, error, warn, info, debug) (default "warn")
      --nonblocking          When set, rp-cli never waits for events on the websocket (builds, tests, etc.)
      --rawoutput            When set, rp-cli output only machine readable text (json) that corresponds to the Redpesk backend answers
      --serveralias string   Alias for the connection parameters to use to connect on the Redpesk server (default "default")
  -t, --timestamp            Prefix output with timestamp [NOT IMPLEMENTED YET]
$
# Example of image creation
$ rp-cli images add -n "My beautiful image" --arch x86_64 -d redpesk-lts-arz-1.0-update --security smack -p redpesk-industrial-image \
    --ksurl "https://github.com/redpesk-infra/rp-kickstarts.git?.#arz-1.0" \
    --kickstart "Redpesk-minimal-x86_64-smack.ks" \
    -r "https://download.redpesk.bzh/redpesk-lts/arz-1.0-update/packages/middleware/x86_64/os/" \
    -r "https://download.redpesk.bzh/redpesk-lts/arz-1.0/packages/middleware/x86_64/os/" \
    -r "https://download.redpesk.bzh/redpesk-lts/arz-1.0-update/packages/baseos/x86_64/os/" \
    -r "https://download.redpesk.bzh/redpesk-lts/arz-1.0/packages/baseos/x86_64/os/"
-- Image creation requested by user --
Checking the repositories architecture...	[OK]
Creation of "My beautiful image" image...	[OK]

Once the image is created in the backend, the user can build it locally thanks to the rp-cli images localbuild command.

NB: The following options are optional at the image creation, but mandatory to have at build time: --kickstart and --ksurl (if some kickstart dependencies are in the git repository).

$ rp-cli images localbuild --help
Start building a redpesk image locally.
An object "image" needs to have been created in the backend with all the rights parameters (cf. 'rp-cli images add --help').
A local kickstart file can be specified through the '--local-kickstart' flag.

Usage:
  rp-cli images localbuild <image-ID> [flags]

Aliases:
  localbuild, lclbld

Flags:
  -h, --help                     help for localbuild
      --local-kickstart string   [Optional] Gives the path to a local kickstart file to use to build the image
  -s, --size int                 [Optional] Specify the disk device size in gigabytes (default -1)

# Example of image localbuild
$ rp-cli images localbuild my-beautiful-image

-- Local build of an image requested by user --

Checking that all needed parameters are set...			    [OK]
Checking and preparing the directory for image build...	[OK]

"Image build" directory set-up: /home/armand/Development/local-builder/gitpkgs/image_localbuild_my-beautiful-image

IMAGE BUILD LOGS >>>
kickstart = Redpesk-minimal-x86_64-smack.ks
No spec file found!
Will try to checkout to arz-1.0 (commit_checkout)
Commit checkout arz-1.0 is set as a RefType.BRANCH ref
Trying shallow clone source from https://github.com/redpesk-infra/rp-kickstarts.git:arz-1.0 into /tmp/tmpt26jshzf/gitsource
Successfull shallow cloned.
Preparing the kickstart
Writing kickstart in temporary dir /tmp/tmpt26jshzf/Redpesk-minimal-x86_64-smack.ks
Writing TDL in temporary dir /tmp/tmpt26jshzf/tdl-x86_64.xml
Building image
Running: oz-install -d4 -t 14400 -a /tmp/tmpt26jshzf/Redpesk-minimal-x86_64-smack.ks /tmp/tmpt26jshzf/tdl-x86_64.xml
DEBUG:oz.Guest.RedPeskGuest:libvirt bridge name is virbr0
DEBUG:oz.Guest.RedPeskGuest:Libvirt type is kvm
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): download.redpesk.bzh:443
DEBUG:urllib3.connectionpool:https://download.redpesk.bzh:443 "HEAD /redpesk-lts/arz-1.0/imager-os/x86_64/ HTTP/1.1" 200 0

[..]

DEBUG:oz.ozutil:Waiting for my-beautiful-image to finish installing, 13409/14400
DEBUG:oz.ozutil:Waiting for my-beautiful-image to finish shutdown, 90/90
INFO:oz.Guest.RedPeskGuest:Install of my-beautiful-image succeeded
INFO:oz.Guest.RedPeskGuest:Generate XML for guest my-beautiful-image with bootdev hd
DEBUG:oz.Guest.RedPeskGuest:Generated XML:
<domain type="kvm">
  <name>my-beautiful-image</name>
  <memory>4194304</memory>
  <currentMemory>4194304</currentMemory>
  <uuid>7e684acd-1159-46e1-b85c-c5c1561054df</uuid>
  <clock offset="utc"/>
  <vcpu>4</vcpu>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <os>
    <type machine="q35">hvm</type>
    <boot dev="hd"/>
    <loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram template="/usr/share/OVMF/OVMF_VARS.fd"/>
  </os>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <graphics port="-1" type="vnc"/>
    <interface type="bridge">
      <source bridge="virbr0"/>
      <mac address="52:54:00:c4:8b:af"/>
      <model type="virtio"/>
    </interface>
    <input bus="ps2" type="mouse"/>
    <serial type="pty">
      <target port="0"/>
    </serial>
    <serial type="tcp">
      <source mode="bind" host="127.0.0.1" service="34719"/>
      <protocol type="raw"/>
      <target port="1"/>
    </serial>
    <rng model="virtio">
      <rate bytes="1024" period="1000"/>
      <backend model="random">/dev/random</backend>
    </rng>
    <disk device="disk" type="file">
      <target dev="vda" bus="virtio"/>
      <source file="/home/devel/.oz/images/my-beautiful-image.dsk"/>
      <driver name="qemu" type="raw"/>
    </disk>
  </devices>
</domain>

INFO:oz.Guest.RedPeskGuest:Cleaning up after install
Libvirt XML was written to my-beautiful-imageApr_12_2022-16:04:54
Compressing image
/home/devel/.oz/images/my-beautiful-image.dsk (1/1)
  100 %     599,5 MiB / 7 168,0 MiB = 0,084    41 MiB/s       2:55             
Removing the image and workdir.
Running: rm -rf /tmp/tmpkg133arj /home/devel/.oz/images/my-beautiful-image.dsk my-beautiful-imageApr_12_2022-16:04:54
<<< IMAGE BUILD LOGS

Congratulations! Your image build seems to be successful!
Image available in: /home/devel/sandbox/gitpkgs/image_localbuild_my-beautiful-image