Azure DevOps CI-CD for BizTalk with BTDF and Powershell

In this blog, I would showcase a simple demo to deploy application with BTDF and Powershell scripts using Azure Devops CI-CD pipeline.

In the BizTalk solution, I added the power-shell scripts to the solution as shown below and customized the RunDeployment.ps1 script to be generic by taking in parameters(Application, Deployment Type and Environment),please refer this article for the power-shell scripts.

Refer below screen shot for CI Pipeline configuration. Here, I'm building the BizTalk solution and then with msbuild task we are building the BTDF project. Once we have the msi file, we are copying those binaries to the staging area and then packaging to the Azure Artifacts.

Refer below screen shot for CD Pipeline configuration, here we have added just power-shell task and configured the parameters.

In order to achieve CI-CD, I enabled the continuous integration and delivery options. Now, for every check-in against the master branch, BizTalk application would be deployed in corresponding environment.

Azure DevOps for BizTalk CI-CD

In this blog, I would showcase a simple demo to deploy application using Azure Devops CI-CD pipeline with BizTalk Server Application Project.

In order leverage Azure Devops,  we should setup a build /azure pipeline agent in the environment(Dev,UAT and PROD).

Go to Organisation Setting in AzureDevops portal and click on  download agent from the Agent pools page.  Extract the zip and run the config.cmd. Now, provide the organisation url and access token (PAT) then configure the agent name and work directory as shown below

Once the build agent is ready, let us add the Application Project to the existing BizTalk solution and configure the JSON file, in this file we can define the order of BizTalk assemblies to be built and deployed and specify the bindings to be imported, we can even have the pre and post processing scripts as shown below. 

 Let us create the CI-CD pipeline in Azure Devops, navigate to Build under the pipeline and choose the repos and go an empty template as shown, 

Let us choose the agent and then add tasks to build(msbuild) and publish(publish artifacts) the artifacts in the azure tfs location, this published artifacts will be used by release pipeline for the deployment.

Here we will configure, release pipeline using the output of build pipeline. Create a new release pipeline and configure the build output and select tasks, choose Deploy BizTalk Server application and configure it as shown below.

Now, Build  (CI) and Release(CD) Pipelines are ready, we can enable continuous integration and continuous deployment triggers in build and release pipelines as shown below.

For every changes made to the Repo branch, CI and CD will be triggered and application would be deployed.

Migrating BizTalk Transformation To Azure using Azure Function as an API

In this article, I would brief about Migrating BizTalk Transformation As-Is to Azure Function(API) and this can be invoked from Logic App. Thus the transformation can be execute across different resource group, region and integration account.

Using Azure SDK, I have create a simple HTTP Trigger function. This has two projects, TrasformationHelper(TransformAPI) and AzureFunctionHelper.

In TransformAPI, we extract the request body and map name from the query string and invoke utility helper for the transformation.

In AzureFunctionHelper, I have embedded the XSLT and Extension Objects as resource files and based on the MapName, we would dynamically execute the transformation and render the transformed result as HTTP Response to client.

Here is the XSLT for one of the transformation and this invoke external assembly methods, as shown. External assembly method does DB Lookup. Lets add the reference of External Assembly to Azure Function Project.

Lets configure hybrid connection in Azure Function. select Networking and click on configure your hybrid connection endpoints. Click on Add hybrid Connection, 
Create a new or bind to existing Hybrid Connection. Refer my earlier article for configuring azure hybrid connection to access on-prem sql server.

Lets deploy and test the Azure function API. This can be invoked from Logic App to achieve dynamic transformation.

Note:- External Assembly would be package as part of Azure Function Package. This allows us to completely re-use BizTalk Transformation as-is.

Exposing On-Premise BizTalk Service to Internet with Service Bus & WCF-Relay Adapter

In this article, I would walk through on how to leverage WCF-Relay adapter to expose on-premise BizTalk service for public access.

Refer my earlier article for Boomerang pipeline component, as I would use the same component over here.

Lets create a two way service which receives the request and renders a transformed message. As show below create receive port  and a receive location with WCF-BasicHttpRelay binding, then provide the relay URL and configure the SAS Policy under Bindings Tabs.

Here is simple token provider console application, which generates the access token. This token has to be passed as authentication header in order to access the service.

Lets generate the token and using post man, these test the on-premise BizTalk service by posting a HTTP request with a valid request body.

As shown, transformed response would be rendered back to the client. By this way, we can access our on-premise service over the internet. Key take is that we can leverage BizTalk transformations from Azure Functions/Logic Apps instead of writing custom API to achieve complex transformation involving functiods.

Hybrid Adapters BizTalk Server 2016 | Event Hub & Service Bus Messaging Adapters

In this article, I would brief on configuring BizTalk 2016 Hybrid Adapters like Service Bus Message(Queues) and EventHub. These adapaters can be configured at receive and send ports.

Here, I have a Azure function which would act as a publisher to Event Hub and Service Bus Queue, you can refer my earlier Azure Function blog post.

Lets create a simple messaging solution for our demo purpose; then add and configure EventHub adapter as shown below, By Signing into Azure Account, we can configure the appropriate EventHub instance and its related SAS policy. 

Now, lets configure the SB Messaging Adapter, please provide the SB namespace and the queue name; In authentication tab provide relevant SAS Policy.

Let create a subscription inside BizTalk at the Sendports with ReceivePortName for EventHub and SB Messages; let test and verify the end to end flow.

Optimizing BizTalk Map Execution and Modularizing Map using Import XSLT

Here in this article, I would brief about optimizing map execution using XSL Compile Transformation and achieve Importing XSLT using helper component.

Let go straight into mapping scenarios, below is a use case where we have huge Inline XSLT code, this can be moved to separate xsl file as template and this file should be embedded as resource in BizTalk  assembly. As you see, we have moved the code to separate XSL file and within BizTalk using XSLT Scripting functiod, we would just call the actually template.

There can be a use case, where we have complete custom xsl instead of BizTalk mapper. Here, we have the main custom XSL which invokes the template in other XSL files. Refer below for the child template XSL files. Just to generalize, implementation here, I have created multiple child template XSL files and master template file which just imports these child templates and during runtime, we would import the master template in the BizTalk Map/Custom XSL file using Custom helper component.

Here is use case of multi-part message, we would use a similar approach to earlier one but challenge  is to merge/construct a multi-part message for our execution.

Overview of XSLCompiledTranformHelper, this would be used from orchestrations instead of Transform shape

Critical methods are shown below, if you observe GetMapInfo method, we dynamically import the master template xsl file and EmbeddedResourceResolver resolves imported xsl templates from the assembly.

Here are few critical helper classes.

Here is our orchestration uses XSLCompiledTransformHelper for transformation as shown below, which executes maps for different use cases mentioned in our problem statement.

Multiple part message can be executed either by combining messages into single message and executing the transform or by directly invoking the transform method with message params, as shown below.

I have referred below articles and added this feature to the XSL Compile Transformation component.

• https://blogs.msdn.microsoft.com/paolos/2010/01/29/how-to-boost-message-transformations-using-the-xslcompiledtransform-class/
• https://blogs.msdn.microsoft.com/paolos/2010/04/08/how-to-boost-message-transformations-using-the-xslcompiledtransform-class-extended/

BizTalk Server 2016 | Application Insight Logging | Azure-BizTalk

In this article, I will explain the steps involved in sending BizTalk tracking data/trace details to Azure Application Insights.

Lets create an instance of Application Insights in Azure portal as shown below, 

• Log on to Azure portal and add a new application insights instance and Provide an appropriate name. 
• Choose application type as General. 

Make sure you Download the BizTalk Server 2016 Feature Pack 1; this update enables BizTalk to integrate with Azure Application Insight. Once the feature pack is installed, we can see a section called “Analytics” in BizTalk Group hub settings page.

Once the analytics is configured at the group level, now we can enabling tracking at individual artifacts level, as shown below.

I have enabled the Analytics for send port, receive and orchestration. This ensures that all the data will be pushed to Application insights. From Application Insight azure portal; You can explore these metrics and create graphs for data visualization; once the data is pumped to Application Insights.

Below are few pointers on AppInsight logging mechanism for BizTalk. 

• Captures analytics on BizTalk Message Box only.
• Context Properties available on messages are captured in App Insight.
• For Physical Ports 2 entries are made (Receive and Send | Publish Subscriber)
• For each logical Port, an entry is made (Orchestration)
• There should be a dedicated Send Port for routing exception message to capture exceptiondetails in App Insight.
• Analytics can be captured at Ports and Orchestrations. It’s not mandate to enable tracking at ports and orchestrations; enabling analytics is sufficient.
• Orchestration entries are captured with respect to logical ports. If call or start orchestration doesn’t have any logical ports; analytics entries for these orchestration aren’t captured.

Here is my simple helper class using this we could capture the trace details from orchestration.

This is my simple orchestration, which would send the trace logs to application insight using the helper class.

As you can see the logs are available under traces.

BizTalk Deployment | BTDF Framework

In this tutorial, I would showcase a simple demo to deploy application using BTDF framework. BizTalk Deployment Framework (BTDF). This open source framework is often used to create installation packages for BizTalk Applications. Downloaded and installed BTDF on your box.


To the existing BizTalk project add Add a New Project under BizTalk Projects and select Deployment Framework for BizTalk Project, as shown

In Deployment Framework screen configure the artefacts you want to deploy and click the Create Project button, now the project file is created. Add  all the relevant files manually to the Deployment solution Folder.

Add  all the relevant files manually to the Deployment solution Folder. We have the Deployment project in place, add basic BizTalk artifacts to the Deployment project.

An ItemGroup is an element from MSBuild which is used by BTDF. Besides ItemGroups, you will also find PropertyGroups within the BTDF project file. PropertyGroups are used to organize all kind of parameters which can be used during deployment.

Add schemas, maps and orchestrations to the deployment project, follow below steps and repeat the same for each artifacts under respective element tags schema, transform and orchestraion.

In the Deployment project file, search for the ItemGroup that is used for deploying Schemas. It contains the text ‘<Schemas Include=”YourSchemas.dll”>’.

Replace ‘YourSchemas.dll’ with the name of the DLL which contains your schemas

The line with the ‘<LocationPath>’ tag, must contain the path where the DLL resides.

Now deploy the solution from VS as shown, but we see an error as it couldn't enlist the orchestration.

Open the BizTalk Server Administration Console and navigate to the application and bind the orchestration and logical ports. And export the bindings to generate master portbinding file.

Open SettingsFileGenerator.xml and add the environment related settings for each environment as shown.

For example use sendfilepath variable from the settings file in the PortBindingsMaster, you need to open the PortBindingsMasterFile.xml file, navigate to the Send Port which should use the addresses and replace the current address with ${sendfilepath}. Refer below screen shot.

Post setting up the binding, lets deploy the application from Tools-->Deployment Framework for BizTalk-->Deploy BizTalk Solution

BizTalk Server Video Tutorials for Beginners

Understand the Basic concepts in BizTalk, here is the video playlist to my channel.

Boomerang BizTalk Web Service for Transformation

In this article, I would show how to develop a Transformation Web Service in BizTalk with a simple custom pipeline component. This can be extended to leverage other BizTalk features.

Lets create custom pipeline component as shown below, this a simple component which writes the incoming property back to the RouteDirectTo and EPMRRCorrelationToken context properties, which would help us to achieve the echo back service or boomerang service.

Lets create a BizTalk project, add Source and Destination schema then add maps to do simple direct mapping from SourceA To DestinationA and SourceB To DestinationB. Finally, add receive pipeline and have XML Disassembler and our custom boomerang component as shown.

Lets deploy and configure the Application. Here we would just add a simple Two-way receive port and configure the receive location as shown below and configure the maps at the inbound map in the receive port.

Test and verify the process, refer below for execution. As you post the source message, pipeline at dissemble stage context properties are promoted and when it comes to resolver stage, boomerang component rewrites the context properties in such as way, it is subscribed the receive ports send side. Thus the transformed message are echoed back. Similar to boomerang!!  

Executing BizTalk Pipelines Inside Orchestration

In this walk through, I brief about how to execute the pipelines inside orchestration,
add references of Microsoft.XLANGs.Pipeline.dll and Microsoft.BizTalk.Pipeline.dll assemblies. In my earlier article (Custom Pipeline and Pipeline Component), Wrapper Order Request message was debatched in BizTalk Receive Pipeline using Custom Detacher Component. The same detaching logic is implemented inside orchestration.  Below process debatches the received message using SnippetDebatchRP receive pipeline, make orchestration transaction as atomic in case if we are executing receive pipeline.

Below process aggregates the message based on OrderID using correlation and as we are executing send pipeline inside orchestration and the process is of none transaction.

Encryption and Decryption of Sensitive Information using Pipeline Components in BizTalk

In this article, I will brief on encrypting and decrypting the sensitive information using custom pipeline component in BizTalk.

Lets create a class library project to implement Execute pipeline component as shown below, relevant reference are added to develop custom component.

Now, lets define properties to specify the cryption type (encryption or decryption) on data and data node as shown. Cryption Type is defined as enum type as below.

Here is the implementation of  IPersistPropertyBag interface, for CryptionType and Element Properties.

Implementation of IComponent Interface is shown below, based on the CryptionType property and Element Name specify, the data would be encrypted or decrypted.

Internal Encryption and Decryption implementation,

Lets create encryption receive pipeline and bring in EncryptDecrypt Component to Decode stage and define the properties as below.

Similarly, Send Pipeline to Decrypt the information

Deploy the solution and define a receive port and location then associate the EncryptData Receive Pipeline as shown below.

Similarly, define two send ports to subscribe the message received on the receive, 1 send port will have DecryptData Send Pipeline and other would be a simple pass through pipeline.

Pass through Send Pipeline, subscribes the encrypted message.

Process messages and verify the result. 

This mechanism allow us to transfer sensitive information in a is a secured manner and thus 3rd party system would be able to access the information.