Getting started with CI/CD using Delphi and Jenkins on Windows

Updated: Feb 2, 2021

Many Delphi programmers are working on "legacy code" which has little or no tests and is often built and deployed manually. Unfortunately, many Delphi developers are stuck in this rut and could greatly benefit by upgrading their overall dev process. For many large projects, it is dismissed as being "too difficult" or "too risky" to even attempt to start the needed cleanup process. This reminds me of the question "How do you eat an Elephant?" and the proper response "One bite at a time!" Nothing is too difficult if you break it down into small bite size steps! That is exactly what we will attempt to demonstrate in this article as we'll get a Jenkins server installed, a Delphi build machine created, fork a sample GitHub repository and automate builds and unit tests.

One of the first "bites" at solving this overall tooling issue is setting up your first build server. The most utilized build server today is Jenkins. There are other very popular options out there, and I plan to cover a few in upcoming articles, but Jenkins is still the king.

Jenkins with Delphi logo

Jenkins has an automated setup process for Windows so you can get Jenkins installed in just a few minutes. Jenkins is also a Java application so it can be installed on nearly any platform. There is an enormous amount of documentation on Jenkins available, a huge community for asking questions, and it is just a natural first step to take to improve your development process. Since legacy Delphi developers are all using Microsoft Windows, we will cover the setup of Jenkins on Windows in this blog post.

Some Automation Topics

We will first quickly cover some general automation topics and later delve into the actual process of installing and utilizing Jenkins.

CI/CD and Jenkins

Before installing Jenkins, exactly what is it? I tend to reference it as build server, but it is much more than just that - Jenkins is an automation server as it can build, test, deploy, and automate much of the tasks around any project. It is also referred to as a CI Server for its ability to handle Continuous Integration tasks. This is the initial "build server" activity where the code base is built and tested automatically based on check-ins and/or on a scheduled basis. It is also referred to as a CD Server for Continuous Delivery or Continuous Deployment which are manual or automated deployments. These terms are generally combined as CI/CD as the goal is to automate as much of the overall process as you can. You utilize CI to build the artifacts deployed via CD. The building, testing, deployment, and monitoring of your applications are what make up your DevOps pipeline. Jenkins can be configured to handle simple builds or expanded to handle more complex pipelines. If you are just starting out with Jenkins, it is best to utilize simple build jobs and then migrate later to pipelines.

Trunk Based Development

If you are just starting out with CI/CD, then I highly recommend considering Trunk Base Development with only short-lived branches. This will simply your overall process and free you from merge hell. Many companies successfully utilize long running branches and still have a well-defined CI/CD process, but it is a little more complicated to pull off. Regardless of your branching strategy, a well-defined process for committing changes is required before automation can be successful. Formalize the process with a written document of understanding for all developers involved. An updated (or brand new) set of documentation targeted at your developers will be valuable when you start automating your builds and you should continually add to it as you improve your system.

Continuous Integration for Those Without Any Tests

If you are just starting out with Jenkins, and do not have any tests at all, you can still greatly benefit from Continuous Integration. In many shops, just getting the code to build on a regular basis is a large step forward. If you are in this position, start with the smallest project available and figure out what is needed to build that smallest of projects and add it to your documentation and test building it manually. You can then use Jenkins to automate the process so it is built periodically (at least once/day for most applications.)

The reasons may not be obvious just yet, but if you can get one project built automatically then you will get quicker feedback when someone eventually checks in code that breaks the build. The main fact for everyone to acknowledge is that, in general, the quicker you discover problems, the cheaper they are to fix.

Establish Command Line Builds for Every Project

If you are supporting an ancient product, you may be surprised to find how many different source code projects that you maintain! For every source code project that you support, you need to figure out how to build the project from the command line. Some projects may be only built from certain machines so you should document all project requirements and dependencies in a master list.

Once you can easily build a project, then you can quickly create a Jenkins job to have Jenkins execute that command line build. The next step would be to schedule that job to be executed automatically on a nightly basis and if any errors occur during the build you can send a notification message to the group regarding the failure. You will need to define the specific actions which are required to respond to build failures and update your developer documentation accordingly. It is highly recommended to immediately fix any build failures (due to the cost savings mentioned earlier.)

Once you get your first project built automatically, keep slowly adding projects to Jenkins until you have a comfortable grasp of how long your builds take and the availability of build machines. (Eat that elephant one bite at a time!) Eventually, all your projects will be built automatically every night (or how ever often you need to space out your builds based on build-machine availability and other project requirements.) You should celebrate once you achieve this major milestone! At this point, you will have successfully eaten a nice chunk of that elephant!

Automate Your Tests with Jenkins

Once you have a project being built automatically, you should introduce unit and integration tests as an additional code quality validation step. Ideally, you will end up with a suite of tests for every source code project being built. Once you have the tests built, and error notifications being sent, continue to update your developer documentation to define the actions to be taken when an error is received. TIP: If possible, define the actions to be taken within the notification message itself so the action to be taken is obvious.

You can spend a considerable amount of time creating tests for every project that you manage. The initial goal is to verify basic functionality. If you are just starting this process, don't get trapped into the mindset that you need to create tests to validate every single line of code. Again, take a small bite - as getting a single test in place is far better than no tests at all. If you already have plenty of unit and integration tests, perhaps think about performance and load tests. There are a wide variety of testing techniques that you should consider adding to your automation process.

Tests are often broken down into Functional and Non-Functional types. Always start with the Functional Tests to "ensure it works" and then introduce related Non-Functional Tests to test the overall readiness of use. For example, if your software produces a report, first ensure the report properly displays the correct output. If your report can be run by at most 15 users at the same time, then develop a load test to ensure that at least 15 concurrent requests can be handled at once on hardware similar to your production equipment (while still displaying the correct output - do not let your performance tweaking kill your basic functionality.)

Getting Started with Jenkins and Delphi

Now that some of the basic information has been reviewed, it is time to dig into a little more detail with some screenshots and getting Jenkins running!

Decide on LTS or Weekly Updates

Jenkins follows a common trend in Enterprise Software which offers a Long Term Support (LTS) release version and other more frequently updated versions. There are many companies that prefer stability over new features so they would rather manage fewer updates. In you are in that group, you would select the stable LTS release of Jenkins. If you would like to get the latest and greatest that Jenkins has to offer, you will want to select the Weekly Release version.

Download Jenkins And Review Requirements

The latest LTS or Weekly Release of Jenkins is available on the Jenkins download page. There is a wide variety of options to select from on this download page. You can get installers for multiple platforms, get a Docker container image, or simply download the .war package files. They also provide links for deploying Jenkins in the cloud.

While you are on the download page, check out the Hardware and Software requirements for Jenkins. Note that since Jenkins 2.238 only .NET Framework 4.0 or above is supported for Windows service installations. (Older releases supported .NET Framework 2.0)

Also note the Browser Compatibility Page for Jenkins which lists Chrome, Firefox, Safari, and IE 11 as the main Level 1 targets as the Jenkins team proactively address issues for these browsers. Microsoft Edge is a Level 2 target which means it is a "best effort" situation to solve problems with that browser.

Finally, review the Java Requirements Page for Jenkins which lists Java 9, 10, 12 as not supported. They perform a full test flow for OpenJDK JDK / JRE 8 and 11 (64 bit) so these would be the suggested Java versions to target. You will get a warning if you run the controller or agent with an unsupported version. I use the AdoptOpenJDK installer for OpenJDK 11 (LTS) with HotSpot JVM for Windows x64. Note that some Jenkins documentation references Java Web Start and this technology was removed from Java distributions starting with Java 11. See a StackOverflow related question on the status of this. You will need to keep this in mind if you see Jenkins documentation referencing "JNLP" type URLs which auto-launch Java applets as that will no longer work as expected.

Jenkins is also available for easy installation via Chocalatey if you use that tool, but ensure that you review a currently open issue about upgrade failures with the latest package.

Decide Where to Install Jenkins server

Many developers simply install Jenkins on their main development machine for their own personal use and they launch the Windows Service utilizing their own Windows username so the Jenkins service has access to all their installed software. For dev teams, you will likely want to install Jenkins on a stand-alone server and install agents on build machines for distributed builds. You may also want to consider a cloud based installation. Many will all need to consider high availability options for Jenkins, which is typically solved by using a High Availability plugin from CloudBees.

Remember that if you will be using Jenkins with agents on different build machines, you will need to open a port on the firewall of the server running the Jenkins main controller. There is an option in the Installation process to automatically create a Windows Firewall rule that you can see in the screen shots below.

As a reminder for wherever you install Jenkins, you will want to ensure that you are backing up your Jenkins "Home" folder which contains your build job configurations and build history. There are a few plugins that can help with this task but be aware that some of the commonly referenced plugins have not been updated in a while.

Install Java

Before installing Jenkins, ensure your target server has Java installed as you will not be able to finish the install process without Java being detected on the system.

Note when installing Java via the AdoptOpenJDK installer select the option to "Set JAVA_HOME variable" so the Jenkins installer can find java.exe as this option is not selected by default.

Set JAVA_HOME when installing JRE or JDK

You can easily verify if Java is installed by opening a command prompt and running the command to view the version number:

java -version

To verify the JAVA_HOME variable has been set, at a command prompt use the following command to view the current setting:

echo %JAVA_HOME%

Install Jenkins On Windows

The Windows installer (LTS or Weekly Release) is jenkins.msi and it involves only a few screens and typically takes just a minute or two to complete. Once the Windows installer finishes, the installer will launch a web browser to connect to Jenkins and complete the installation process.

When the installer is launched, a simple welcome screen is presented to start the setup wizard. Verify the release number displayed and click on Next to continue. (I am currently using Jenkins release 2.275 which is the latest Weekly Release as of today.)

Jenkins Installer Wizard on Windows welcome screen

The next step in the install wizard process is to select the destination folder for the Jenkins application files (which includes jenkins.exe, Jenkins.war, and jenkins.xml. There will also be a few log files created in this folder.) Note that the main data files will be stored in a different folder based on the user that launches the Windows Service. This application folder contains less than 70 MB worth of data so you can normally keep the default folder unchanged: C:\Program Files\Jenkins and click on Next to continue.

Jenkins Installer Wizard destination folder

Next, you need to define which user will launch Jenkins as a Windows Service. You can run the service as LocalSystem or enter in a specific Windows account and password. If you are going to run build jobs on the main controller machine, this account needs access to the tools required to build your applications. For example, if you are installing Jenkins on your Delphi machine, input your own username and password to utilize your copy of Delphi on the machine.

If you do not need to run build jobs on this machine, you could easily select the option to "Run service as LocalSystem" to run the Windows Service - just note that the Jenkins data files will be stored in C:\Windows\System32\config\systemprofile\AppData\Local\Jenkins by default which is typically not a good idea. It is usually a better idea to utilize a dedicated user account for launching your service.

Jenkins Installer Wizard service logon credentials

If input a specific user account, then that account needs to have Logon as a Service privilege assigned to it. The installer wizard requires you to click the Test Credentials button before continuing and you will get an error if the selected account fails to pass this validation test.

Invalid logon due to missing required privilege

You can ask your domain administrator to do this, or if you want to change this locally in Windows 10, run the Local Security Policy application and in User Rights Assignment under Local Policies, find "Log on as a service" and double-click it to bring up the list of users allowed. Click the "Add User or Group..." button to add your user account.

Local Security Policy - log on as a service

Once your user account as the Logon as a Service policy applied, click on the Test Credentials button in the Jenkins setup and it should now show a green checkmark on success and you can hit Next to continue.

Logon credentials validated

The next choice to make is the webserver port number that the Jenkins controller will utilize and it defaults to port 8080. You will need to click the Test Port button to ensure that the port number is not already in use before continuing with the installation wizard by clicking on Next.

Jenkins Installer Wizard port selection

Next, you will need to select the Java path which Jenkins will use. The installer automatically searches the system and defaults the path for you if Java is found. Note that if Java is not found, ensure the JAVA_HOME environment variable is defined. Verify the desired path and click on Next.

Jenkins Installer Wizard java home directory selection

Finally, you can optionally have the installer configure the Windows firewall to allow inbound connections to the port number which you selected earlier. This will create a new firewall rule named "Jenkins" allowing connections to the configured java.exe program for inbound traffic on your configured port.

Jenkins Installer Wizard custom setup for adding exception to firewall

On Windows 10, you can review and edit this new Inbound Rule in the Windows Defender Firewall with Advanced Security application. (Note that I had to change the new rule's "Program and Services" configuration from the defined java.exe program to "All programs that meet the specified conditions" as my remote Agents could not connect until this firewall rule was modified and the Jenkins service restarted.)

Windows Defender Firewall jenkins rule configuration

The installer is now ready to begin the Jenkins installation! After clicking on the Install button, UAC will prompt you to allow it to continue and you will be able to verify the publisher certificate.

Jenkins Installer Wizard ready to install

After a very quick install process (taking a few seconds on my machine), you will be prompted to Finish the Windows installation wizard.

Jenkins Installer Wizard completed

After the initial wizard closes, your default browser will appear displaying the Jenkins "Getting Started" browser-based wizard to finish the installation process. On slower machines, this may take a minute or so before the Jenkins system is ready to continue. You can restart this browser session by visiting http://localhost:8080 (or whatever port number you configured in the setup process.)

The first step in the browser wizard is to Unlock Jenkins. This is a security check to ensure that whoever is running the browser wizard has full access to the local system. You will be provided with the full path of the initialAdminPassword file and you will need to input the contents of that file and then click on Continue. The contents of the file is a single line of text like e83877c976f44fbf9c959c85c8a5f591. Note this is also the assigned password for the Jenkins "admin" user account, so keep it secret, keep it safe.

Jenkins Browser Setup Wizard - unlock jenkins

You are then given two options to Customize Jenkins. You can install the recommended set of plugins or manually select which plugins are installed. Click on the second option to customize the plugins as we will be adding a couple of plugins for our use with Delphi.

Jenkins Browser Setup Wizard - customize jenkins

A list of plugins is displayed with the recommended plugins already pre-selected. Scroll down and select the xUnit plugin. Also review the Source Code Management plugin options and select your desired version control plugins. If you are going to follow along and use the samples provided, ensure the Git plugin is also selected. Click on the Install button when you are ready to continue the process of configuring the Jenkins server.

Note that you can always add or remove plugins later by using the Manage Plugins menu option within the Manage Jenkins screen available from the Jenkins Dashboard.

Jenkins Browser Setup Wizard - configure plugins

The browser wizard will install all selected plugins before continuing. This will take a little time to complete (on my system it took about 2 minutes, but it depends on the number of plugins you are installing and their required dependencies.)

Jenkins Browser Setup Wizard - installing plugins

You will then be presented with the option to Create First Admin User. Enter in your desired username and password along with full name and email address. When ready, click on the Save and Continue button. Note that you can skip this step if you want to continue and use the default admin account and setup other users later.

Jenkins Browser Setup Wizard - create first admin user

Finally, you will be prompted to modify the Instance Configuration. If you have a custom host name for the server Jenkins is being installed on, enter it here. At minimum, change the default localhost to the IP address of the server if you are going to be accessing it remotely. Click on the Save and Finish button when done.

Jenkins Browser Setup Wizard - instance configuration

Jenkins is now setup and ready to be used! Click on the Start using Jenkins button to complete the browser wizard.

Jenkins Browser Setup Wizard - Jenkins is ready

You should be presented with the main Jenkins Dashboard. You can access this at any time using the URL defined in the Instance Configuration. You will be prompted to logon with your username and password as needed when revisiting the server.

Jenkins Dashboard

Note: if you will be using Git in your jobs, ensure you that Git For Windows is also installed on your Jenkins server.

Configure A Delphi Build Agent

You can install an Agent on the machine that you use Delphi on today, or you can create a dedicated build machine which we will do for this example. (Skip this step if you want to use your currently configured Delphi machine to act as your build agent.) Note that your Delphi license allows you to copy these files to a separate computer only for the sole use of performing automated builds.

Copy Delphi Files to The Build Machine

On our target build machine, create a folder called C:\Delphi10.4. Copy the following folders from the machine where Delphi is installed to this target folder (excluding platforms that you do not need on the build agent if desired.)

  • C:\Program Files (x86)\Embarcadero\Studio\21.0\bin

  • C:\Program Files (x86)\Embarcadero\Studio\21.0\bin64

  • C:\Program Files (x86)\Embarcadero\Studio\21.0\lib

Also make a copy of EnvOptions.proj from %APPDATA%\Roaming\Embarcadero\BDS\21.0

Customize Build Machine Paths

Edit C:\Delphi10.4\bin\rsvars.bat and change the paths to match the paths on this build machine, something like the following example:

@SET BDS=C:\Delphi10.4
@SET BDSLIB=C:\Delphi10.4\Lib
@SET BDSINCLUDE=C:\Delphi10.4\include
@SET BDSCOMMONDIR=C:\Delphi10.4\public
@SET FrameworkDir=C:\Windows\Microsoft.NET\Framework\v4.0.30319
@SET FrameworkVersion=v4.5
@SET FrameworkSDKDir=
@SET PATH=%FrameworkDir%;%FrameworkSDKDir%;C:\Delphi10.4\bin;C:\Delphi10.4\bin64;C:\Delphi10.4\cmake;%PATH%
@SET PlatformSDK=

Copy the EnvOptions.proj file to the APPDATA folder of the user account which will execute the builds. (For example: C:\Users\JenkinsUserName\Roaming\Embarcadero\BDS\21.0) If you have custom paths for libraries and component packages, edit the DelphiLibraryPath for each target platform that you will use to match your build machine paths. You will get a warning message in your builds if this file is not found, something like the following:

Expected configuration file missing - C:\Users\YourName\AppData\Roaming\Embarcadero\BDS\21.0\EnvOptions.proj

Ensure Prerequisites Are Available

Since we are typically installing only on Windows 10 today, you should not need to install the .NET 4 Framework as MSBuild.exe will be available. You do need to ensure Java is installed, and you can use the same AdoptOpenJDK installer as used above. Also, for our example project demo, Git will be needed by the Agent to check out files from GitHub, so install Git For Windows as required.

Include Third Party Components

Managing the components used by applications can be a chore in Delphi. While it can be simple on the build machine to copy the needed source and BPLs for third party components, some may need special installation and licensing requirements. Proper component management is an entirely different elephant to eat (and for a different blog article.)

Note: if you will be using the command line compiler to build Delphi projects, ensure you update the dcc32/dcc64.cfg files in the Delphi10.4\Bin folder as stated within the dedicated build machine link above.

Define a New Node in Jenkins

Note: if you have installed the Jenkins service directly on your Delphi machine, you can skip this step as you will not be needing distributed builds.

Now that we have a Jenkins automation server up and running, lets create an Agent node for building our Delphi projects. From the main Dashboard in Jenkins, select the Manage Jenkins option to bring up the various configuration screens as shown below.

Manage Jenkins page

While in the Manage Jenkins screen, click on the Manage Nodes and Clouds option under the System Configuration section. This will initially list only the master controller node as shown below.

Jenkins Manage Nodes and Clouds screen

Click on the New Node option to define a node for our first Delphi build agent. You will initially be prompted to enter a Node name. Typically, you would use the Windows computer name within the Node name element, but it is up to you to define the convention you want to use. For this example, we will use a Node name of Delphi1. You will need to ensure that each Node name is unique for this server. (Note there are some naming restrictions as you cannot use any characters from this list: ?*/\%!@#$^&|<>[]:;) Select the Permanent Agent option and then click the OK button to continue.

Define new node name