BlueDolphin ODATA API with Python and Pandas

Introduction

This is the first tutorial in a small series on programmatic access to BlueDolphin's data and functionality. The goal is to show how to use BlueDolphin's ODATA API to retrieve data from Python and it explains how to perform basic analysis of BlueDolphin object data with the Pandas library.

Prepare your Python environment

This tutorial assumes that you have a functional (virtual) Python environment, with Python version 3.7 or higher. Make sure that at least the following Python packages are installed. Please note that, depending on your environment, there may be additional packages that require installation.

• The requests package is the de facto standard for making HTTP requests in Python.
• The pandas package provides Data Analysis capability to Python. According to Wes McKinney, the creator of Pandas, its name is derived from 'panel data' which refers to multi-dimensional structured data sets. Conceptually, it offers capabilities similar to the R language.
• The matplotlib package is a comprehensive library for creating static, animated, and interactive visualizations in Python.

The following statements are an example of how to install these Python packages in a Windows environment. Installation for a Linux environment will be similar.

py -m pip install pip --upgrade
py -m pip install requests
py -m pip install pandas
py -m pip install matplotlib

Process and visualize the data

Import the required modules

Configure the API credentials to use

The credentials for the ODATA API are shown in the General Information page of the BlueDolphin user interface, which you can find by navigating the menus Admin → System → General. On that page, the username to use is shown as Database (which is also your tenant name) and the password is shown as Data Collection API Key.

Request BlueDolphin objects using the ODATA API

The following code fragment calls the ODATA API, retrieves all objects in the BlueDolphin repository and stores the response in the response variable.

The response JSON looks like the following fragment, with the relevant data contained in the value array:

{
  "@odata.context": "https://martijnschiedon.odata.bluedolphin.valueblue.nl/$metadata#Objects(ID,Title,Definition,Category)",
  "value": [
    {
      "ID": "568a7057bb1d7106c8b887fe",
      "Title": "Bedrijfsproces",
      "Definition": "Bedrijfsproces",
      "Category": "Bedrijfslaag"
    },
    {
      "ID": "5db71b1bcf15092a5493f3f7",
      "Title": "Verkoop product",
      "Definition": "Bedrijfsproces",
      "Category": "Bedrijfslaag"
    },
    ...
    {
      "ID": "5ffc6d74ad3fbf340417361e",
      "Title": "Voertuig",
      "Definition": "Bedrijfsobject",
      "Category": "Bedrijfslaag"
    }
  ]
}

Please note that:

• Only the specified fields ID, Title, Definition and Category will be queried and returned in the response. This is more efficient.
• The API connection uses the new URL format which now includes the tenant's name (<tenant>.odata.bluedolphin.valueblue.nl). This is useful when working with multiple tenants and stored credentials in applications such as Power BI or Excel, e.g. supporting simultaneous connections to both test and production environments.

For more information about the ODATA 2.0 URI conventions, see this article.

Normalize the JSON response and create a Pandas DataFrame

The following code fragment converts the API's JSON response into a Pandas DataFrame object, so that it can be manipulated further.

Show the categories of BlueDolphin objects in use

We select distinct category values from the list of objects. These category relates to the ArchiMate layer or extension the object is a member of.

Clean up each object's category description

Note that the results of the previous statement returned the category values in the Dutch language, because the language setting of the BlueDolphin tenant is set to Dutch. Therefore, we will map and standardize these category values according to the following table:

Sort the data

Note that sorting the data is not required and will take considerable time with large data sets. However, for this tutorial it results in a more consistent experience in places where data samples are shown. The chosen sort order is Standardized Category, Definition and Title.

Create a statistics DataFrame with the total number of objects in each standardized category

Visualize the total number of BlueDolphin objects per standardized category

Create the object_count_per_definition DataFrame

This DataFrame is created through grouping the data by the Standardized Category and Definition fields, where the field Object Count contains the number of BlueDolphin objects in the repository per each object definition.

The Standardized Category column needs to be included in the group because the column's values need to survive the aggregation performed by count(). Simply put, the column should keep its original values.

Please note that, unlike the previous grouping example, the columns used for grouping the data are not converted to the DataFrame's index. Occasionally, this makes using functions like DataFrame.pivot() simpler, but is often not necessary.

Visualize the number of BlueDolphin objects per object definition

This chart shows the number of BlueDolphin objects for each object definition as horizontal bars, with the largest bar at the top and the smallest bar at the bottom. It also shows the standardized category in the colors similar to those used in the ArchiMate specification.

Please note that the horizontal axis uses a logarithmic scale.

Conclusion

This tutorial has introduced the basics of data analysis with BlueDolphin's ODATA API, Python and Pandas. With the information provided, you can start discovering your own applications of this incredibly useful feature. Since the ODATA API also supports retrieving Relation-data from the BlueDolphin architecture repository, analyzing the relations and related objects might very well be the subject of a future tutorial.

Happy programming!