The purpose of this blog is to explain how SAP DWC task chain can be scheduled or executed from SAP BTP cloud foundry environment using Node JS, so that SAP BTP can be used as a universal scheduling platform.

SAP DWC Task Chain:

SAP Task Chain is used to execute series of task like Table Replication, View Persistency and Data Flow runs. We can compare SAP DWC Task Chain with SAP BW Process Chain.

1. In this blog Sample Task Chain to persist view will be created.
   
2. That task chain will be executed from SAP BTP Cloud Foundry environment using Node JS.
   
3. For Authentication purpose Oauth 2.0 will be used

SAP Data Warehouse Cloud provides a public OData API to execute APIs inside DWC.

Here OAuth2.0 authorization with grant type Authorization Code is being used to authenticate request from Node JS.

Technical Details:

Let’s first create a simple Task chain which will persist data of a remote table.

Let’s find out the backend executable link generated while executing Task Chain in DWC.

Open inspect from browser before executing Task chain. Open network tab as shown in below screenshot.

Now hit Execute. As shown in below screenshot request url “*/start” is being used in backend to execute task chain in SAP DWC.

Now we will execute this “*/start” URL from node js to execute task chain.

Technical set-up:

In SAP Data Warehouse Cloud, create an OAuth Client with the following specifications:

Purpose: Interactive Usage

Redirect URI: We will discuss about this URL later

Note down OAuth Client ID and Secret code, we will use this information in Node JS to execute Task Chain.

Note down below authorization URL and Token URL from SAP DWC App Integration

Now let’s create Node JS project to execute Task Chain

Passing Authorization URL, Token URL, ClientID, Client Secret & Call back URL from SAP DWC as mentioned earlier to get access token from SAP DWC.

Lets try to post “*/start” with above information in Postman to check if this process is working.

Go to Postman and enter Authorization URL, Token URL, ClientID, Client Secret & Call back URL.

After entering above information click Get New Access Token

After successful authentication clink Use Token. Then click on Send

Status Code 202 – Job successfully executed, check SAP DWC integration monitor if the Task chain is running.

Task Chain ran successfully in DWC.

Let’s get back to Node JS code.

Passing access token for validation of request.

Here is the main code to Post start URL with Access Token and Cookies generated from above code.

Now let’s Push this code in SAP BTP Cloud Foundry.

App successfully deployed in Cloud Foundry.

Now go to SAP BTP Cloud Foundry and get the URL of the deployed APP.

Now take the URL and add “/auth/callback”, this will be redirect URL. We have mentioned same URL in Node JS as shown in below screenshot.

Put this URL in OAuth client Redirect URL section in SAP DWC.

Now execute App URL from SAP BTP with “/auth” added at the end.

Lets check Integration Monitor in SAP DWC. Task Chain started running.

We can schedule this node JS application periodically from SAP BTP Job Scheduling Service.

To execute failed Task Chain replace “start” with “/retry” in the execution URL.

Appendix 3: All Project Files

In this appendix you can find all files required to run the described sample application.

manifest.yml

---
applications:
- name: TaskChainRun
  random-route: true
  buildpack: nodejs_buildpack
  memory: 512M
  command: node index

package.json

{
  "name": "node",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "dependencies": {
    "axios": "^1.2.2",
    "cookie-session": "^2.0.0",
    "express": "^4.18.2",
    "express-session": "^1.17.3",
    "nodemon": "^2.0.20",
    "passport": "^0.5.3",
    "passport-oauth2": "^1.6.1"
  },
  "devDependencies": {},
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

index.js

const express = require("express");
const axios = require("axios");
const passport = require("passport");
const OAuth2Strategy = require("passport-oauth2");
var cookieSession = require("cookie-session");
var http = require("http");
const app = express();
const httpServer = http.createServer(app);

// Configure the OAuth2 Strategy with your client ID and secret
passport.use(
  new OAuth2Strategy(
    {
      authorizationURL:
        "*/oauth/authorize",
      tokenURL:
        "*/oauth/token",
      clientID: "clientid",
      clientSecret:
        "clientSecret",
      callbackURL: "*/auth/callback",
    },
    function (accessToken, refreshToken, profile, done) {
      // This is where you would look up the user in your database
      // and return the user object
      passport.serializeUser(function (user, done) {
        done(null, user);
      });

      // passport.deserializeUser(function (user, done) {
      //   done(null, user);
      // });
      console.log("Access Token", accessToken, refreshToken, profile);
      return done(null, accessToken);
    }
  )
);

// Configure Express to use sessions and Passport
app.use(passport.initialize());
app.use(
  cookieSession({
    name: "session",
    keys: ["ANYTHING"],
    // Cookie Options
    maxAge: 24 * 60 * 60 * 1000, // 24 hours
  })
);
// Set up the authorization route
app.get("/auth", passport.authenticate("oauth2"));

// Set up the callback route
app.get(
  "/auth/callback",
  passport.authenticate("oauth2", { failureRedirect: "/login" }),
  function (req, res) {
    // Successful authentication, redirect home.
    console.log("rbody", req.user);
    console.log("cookie", req.headers.cookie);
    axios
      .post(
        "*/Task_Chain_1/start",
        {},
        {
          headers: {
            Authorization: "Bearer " + req.user,
            Cookie: req.headers.cookie,
          },
        }
      )
      .then((e) => {
        console.log(e);
      })
      .catch((e) => {
        console.log(e);
      });
  }
);

httpServer.listen(process.env.PORT || 5000);

Using the command line client dis recommended, as it is faster (and you’ll anyways need it, once you have to update a service)

Deploy:

Use cf push from command promt

Logs:

cf logs my App –recent

The post How to run SAP DWC Task Chain from SAP BTP Cloud Foundry appeared first on ERP Q&A.

SAP Intelligent Real Estate is a flexible real estate management solution for the corporate and commercial real estate market that optimizes portfolio performance and occupant experience.

The native integration with SAP S/4HANA and other connected applications allows for streamlined performance of end-to-end real estate business processes in the areas of portfolio, location, space, and contract and lease management. Strategic analytics capabilities are built across products and technologies on top of S/4HANA Cloud for Contract and Lease Management, Finance as well as SAP Cloud for Real Estate on BTP. The dashboard facilitates a sustainable property portfolio management to support better decision-making and drive greater efficiencies.

Highly integrated processes between the architectural view in SAP Cloud for Real Estate and the usage view of properties in SAP S/4HANA Cloud mapped with financial processes provide a new level of transparency and data linkage, harmonized in SAP Data Warehouse Cloud.

SAP provides a comprehensive content package which includes all data models in SAP Data Warehouse Cloud plus the SAP Analytics Cloud dashboard on top of it. It serves for various target groups in the real estate management processes.

This blog is not intended to describe the technical implementation of the solution, but to focus more on the business perspective. What possibilities of evaluation and presentation are given with the content package and how can this content be expanded with the optional data sources and additional information that are available in data sources? Which questions are answered with the dashboard, how can I foresee future trends and forecasts?

Available analytic data sources

With the last release of the DWC/SAC content for 2022 Q4, there is now the third version of the content package for SAP Intelligent Real Estate with new functionality and data sources. The analytic data models in the DWC offers to link and analyze the SAP Intelligent Real Estate Usage data from SAP S/4HANA Cloud together with data from the architecture of the buildings from the SAP Cloud for Real Estate. The following analytic data sources are available in the SAP Data Warehouse Cloud:

• Building Data (SAP Cloud for Real Estate)
  Building Hierarchy, Capacity + Workplaces (per Type) + Area Size (Work- and NonWork-Space), Status down to the level of a space
  
• Occupancy Data (S/4HANA Cloud – Intelligent Real Estate)
  Area not in use, Area in Use, Area Vacant and Area Occupied on Company Code Level
  
• Financial Cost Data (S/4HANA Cloud – Finance)
  Area Information, Cost per Building (operational, capital, administrative, consumption and other cost) based on Company Code (per Cost Center, internal Order, WBS Element)
  
• Financial Actual and Plan Cost (S/4HANA Cloud – Finance)
  Actual and planned costs with calculation of the variance, also considering the cost hierarchy based on Company Code
  
• Contract Data (S/4HANA Cloud – Intelligent Real Estate)
  Contract Master Data and Cash Flow Information (actual and planed records) based on Company Code

Starting from the current year, the information is provided looking back for the last 3 years and looking ahead for the next 3 years. If a different period is required for the data, this can be adjusted at a central point in the data model in the DWC. All currency amounts such as postings in company code currency or contract terms in condition currency are converted into the global currency of the source system.

All of these data sources can be used separately from each other in different dashboards.

There are two additional analytic data sources available in the Content Package that are currently not used in the predefined Dashboard, but these can easily be integrated if this information is needed:

• Land/Site/Parcel-Information
  Information about Land, Land Area, Number of Buildings and Parcels, can be integrated based on Site.
  
• Building Area in Detail
  Area based on Component and Sub-Component, Area in total and separated in Workspace- and Non-Workspace-Area, can be integrated based on space level.

Structure of the dashboard

The dashboard provides the information from the source systems on 6 pages. These are structured as follows:

• Portfolio Overview
  
• 360° Overview
  
• Financial Details
  
• Occupancy Details
  
• Workspace Details
  
• Contract Details

The selection date (Key Date) can be specified via the story filter. When calling up the dashboard, the system current date is always set, but this can be changed by the user to display data from other time periods. It is also possible to set filters for the company code, region, country, site and building. The filter for the planned costs (planning category) can also be set. These filters affect the entire dashboard. The different pages can also be filtered by specific dimensions, e.g. by cost centers, contract types, etc..

In most charts, the data can be displayed hierarchically using the drill-down function. In addition to the cost hierarchy, there is a geographical hierarchy where you can go from the region to the building and sometimes down to the floor/space. In addition, a time hierarchy is also available.

Below are a few examples of how the data in the dashboard is shown:

Portfolio Overview

This tab provides information about the area occupied and area vacant along with the KPI Cost per area unit. In addition, the comparison of planned and actual costs year to date is shown. The total costs are shown year over year. The breakdown of the data can be displayed via hierarchy from the region down to the individual building. The KPIs average cost per area unit, occupancy rate and space vacancy rate are determined on average year over year, including the deviation from the previous year’s value. Measures such as area in use/not in use, as well as the number of sites and buildings and the total cost year over year complete the overview.

360° Overview

The 360° Overview page always provides the view on one site. In addition to the presentation of the costs (actual and plan) and the cost allocation, information about the utilization of the area (occupied and area in use) for the last year is also shown here.

Financial Details

The financial overview is strongly influenced by the cost perspective. In addition to the actual/plan comparison, there is also a representation of the costs according to the cost hierarchy at company code level. Filtering is possible according to various dimensions such as cost center, internal order or WBS Element.

The average costs per area and per workplace are available as key figures. Thresholds in the cost deviation analysis can be adjusted by the user to the company requirements.

Occupancy Details

Here the view of the buildings is related to the area. Which space is occupied and rented, which is vacant and which spaces are not yet allocated. How is my occupancy rate and space vacancy rate related to the selection made? What is my average cost per area unit? Which contracts are active and when do they expire? By selecting the geo-hierarchy, this data can be displayed and analyzed in different granularity from region down to floor/space.

Workspace Details

The focus of this view is the relationship between the available capacity and the assigned workplaces. Where is there still free capacity, what are the average costs per workplace? What is the breakdown of workplaces by workplace type? It is also possible to drill down into the various hierarchical levels to analyze the data from top (per region) to bottom (floor/space).

Contract Details

In the Contract Details view, you can switch to Customer Contracts and Vendor Contracts, and you can also filter by Contract Type. The Contract Value can be viewed on the timeline in the past and into the future, making it easy to identify trends and deviations. The tabular view shows all active contracts with the associated contract area and contract amount, the bar chart next to it can be used to filter contracts that expire in the next 3 years.

Business Questions to be answered

Here are just a few examples of Location and Portfolio Managers questions that can be answered with the dashboard.

• What are my most expensive buildings based on cost per space unit? What is the current deviation compared to the previous year?
  
• How have my costs developed in the last 3 years? Where were there major deviations from cost planning? What are the planned values for the costs for the next year?
  
• Which buildings or which regions have the highest energy consumption measured in terms of costs?
  
• What is the relationship between the available capacity and the assigned workplaces? In which buildings of a site do I still have free capacity? What are my costs per workplace in each country/region/site/building?
  
• What is the ratio of occupied area to vacant area measured by the occupancy rate or the space vacancy rate? Show me the top 10 or less 10 buildings based on these key figures.
  
• Where are there still areas that have not yet been assigned to an Enable Use Group? Show me the top 10 Buildings with area not in use.
  
• Which contracts with which amounts will expire in the next 3 years? Which area will be vacant as a result? How was the contract value based on Customer and Vendor Contracts last year and how is the situation in the next 2 years?

The dashboard supports decision-making based on past, current and future data for various business areas. Deviation analyzes can be used to identify and eliminate weak points.

Conclusion

The dashboard offers a broad view of the real estate business and provides information and key figures for all areas of the company that are important for controlling and influencing. The data sources themselves provide much more information, such as costs according to internal orders or cost centers, contract information at the level of condition types or flow types.

Many things can be customized, such as thresholds or direct links from the dashboard to the contract in S/4HANA Cloud or to the building in Cloud for Real Estate (BTP). The calculation of key figures can also be adjusted company-specifically in the SAP Analytics Cloud. As already mentioned, the data sources can be used independently of each other, so that topic-specific new dashboards with more depth and a higher level of detail can be created. For example, to create a separate contract dashboard which shows information down to the flow type or condition type.

All data models also contain the location dimension for each building, so that all key figures can be displayed on a map:

All in all, the SAP Analytics Cloud dashboard and the data models in SAP Data Warehouse Cloud are a very good basis that covers the core processes of the business with many options for individual extension. The Content Package is available in SAP Data Warehouse Cloud and the SAP Analytic Cloud Content Network since November 11, 2022 and is free to install, customize and use. Extensions and improvements will be made available on a regular basis.

The post SAP S/4HANA Cloud Q4/2022: New Scenario in SAP Data Warehouse Cloud (DWC) provides Analytical Insights on SAP Intelligent Real Estate with SAP Analytics Cloud (SAC) appeared first on ERP Q&A.

How to download and install DBeaver: https://dbeaver.io/download/

The entire process of creating a sample data – from setup to HANA connection and a JSON file generation – including 5 steps with some additional explanations are following:

1. Setting up a database user in SAP Data Warehouse Cloud

• As a first step, you need to have a database user (DB) in SAP Data Warehouse Cloud in order to connect to HANA Cloud in DBeaver. In Space Management, you can create a DB user for data consumption and data ingestion. In this case, you only need to read the sample data views in DBeaver and then export it as a JSON file. As a result, the DB user setup can be done as following:

• Database User Settings:

2. Create a SQL view which consists of a small amount of data (e.g filter on 1000 rows) of the data product and can present how the data product looks like.

• As sample data is just an exemplar of the actual data product so that the data consumer can get a glimpse of what contains in the data product, we only need to select the small amount (some rows) data from the original view which is used as a source for the data product. Create a SQL view as following:

SELECT * FROM “View_1” limit 999

• Then validate SQL to check if there is any error in the SQL statement. Set expose for consumption so that it can be seen in DBeaver.

• Then save and deploy the SQL view.

3. Connectivity with HANA Database in DBeaver

• After creating a DB user in SAP Data Warehouse Cloud, you need to connect to HANA Database using this user.
  
• New Database Connection (the + button on the top left) → HANA → Next

• The Connection Settings should be matched with the information of the DB user:

• Click Finish to connect to the HANA Database.

4. Generate a JSON file for the sample data

• After connecting to HANA Cloud, go the connection you just created on the Tab Database Navigator.
  
• HANA Connection → Schemas → TEST → Views → View_1_Data_Sample → (right click) Export Data

• The next steps should be done as following:

• Note: ‘Directory’ as where you choose to save the exported JSON file and ‘File name patterns’ as the name of the file.

5. Format the JSON file

• In order to read the JSON file in the Data Marketplace, the exported JSON file should be formatted as following:
  
• Open the JSON file in the Text Edit, change the name of the value (in this example “View_1_Data_Sample”) into “data”. Then save.

After finishing the generation process, the JSON file is ready to be uploaded as the sample data for the data product in the Data Marketplace.

The post Creating a sample data for a data product using DBeaver appeared first on ERP Q&A.

Innovate your IT landscape with SAP Data Warehouse Cloud, which is SAP’s strategic target solution for all data warehousing use cases, in line with SAP’s data-to-value portfolio strategy. This blog post provides SAP BW and SAP BW/4HANA customers an overview of how existing on-premises investments can be converted to the cloud. More importantly, a Cloud Transformation Checklist which can be used when preparing the project and during the actual conversion to SAP Data Warehouse Cloud. The checklist contains the various conversion paths (Shell- and Remote Conversion), and their individual steps in different tabs, including the corresponding SAP Notes. The checklist is in no way intended to replace a dedicated project plan, but rather to provide support for effort orientation. For initial project planning and rough structuring, it makes sense to think through all project parts one by one. The checklist is available for download as an attachment to SAP Note 3141688. Tip: Click on the figures of the individual project phases below for better readability.

Overview

SAP Data Warehouse Cloud is SAP’s strategic public cloud product for data warehousing. It improves agility, accelerates time-to-value, and unlocks innovation with unprecedented business user enablement, a new form of openness, and embedded partner offerings. Looking at conversion, SAP offers the possibility of a tool-supported move of SAP BW or SAP BW/4HANA investments to SAP Data Warehouse Cloud, SAP BW bridge (see figure 1). SAP BW bridge is a feature of SAP Data Warehouse Cloud, that provides a path to the public cloud for SAP BW and SAP BW/4HANA customers. It enables connectivity and business content via proven SAP BW-based data integration (extractors) from SAP source systems. In addition, it provides staging layers of SAP BW to manage data loads (including deltas) with partitioning, monitoring, and error handling. This allows customers to leverage their existing SAP BW skills and protect their SAP BW investments in the public cloud.

In any conversion, manual interaction and re-design is required (see Figure 2). The degree of these manual tasks varies from customer to customer and depends on the configuration and state of the SAP BW or SAP BW/4HANA system. SAP has developed tools to automate this renovation where possible and feasible, but they are not built or intended to fix badly designed models or clean up neglected systems. The tools will transfer objects based on XML files via RFC from SAP BW Release 7.30 to 7.50 (on SAP HANA or Any-DB), and SAP BW/4HANA 2021 to SAP Data Warehouse Cloud, SAP BW bridge. Each transfer is based on a selection of a specific scope, i.e., a set of SAP BW objects, e.g., a data flow that can be transferred together in a consistent way. Please note that SAP BW 7.40 and lower are already out of maintenance.

SAP provides a Pre-Check Tool (see SAP Note 2575059) that identifies important steps customers need to take to ensure their system is compatible with the conversion process. The tool determines which objects are compatible with SAP Data Warehouse Cloud, SAP BW bridge and which objects are not available in SAP Data Warehouse Cloud, SAP BW bridge. In addition, it checks which objects can be automatically converted, deleted, or need manual adjustments (see figure 2).

Regardless of the type of conversion (see figure 3), the Simplification List (currently in preparation, see SAP Note 3154420) and the Conversion Guide are suitable as a starting point. The Simplification List describes in detail, on a functional level, what happens to individual data models and objects in SAP Data Warehouse Cloud, SAP BW bridge. The individual Notes explain what needs to be done in the conversion process. The Conversion Guide is the end-to-end documentation of a conversion to SAP Data Warehouse Cloud, SAP BW bridge. Note: There is an individual Conversion Guide for the Shell Conversion (see Link) and an individual Conversion Guide for the Remote Conversion (currently in preparation; Remote Conversion is planned for release 2205).

Shell Conversion to SAP Data Warehouse Cloud

Shell Conversion: General Sequence

Shell Conversion (see figure 4) is offered by SAP to convert an SAP BW or SAP BW/4HANA system into SAP Data Warehouse Cloud. It does not include the transfer and synchronization of existing data sets. Instead, customers can choose to load data from original sources, load data from the sender SAP BW or SAP BW/4HANA system, or simply ignore historical data and start fresh. For SAP BW systems on releases from 7.30 to 7.50 (running on SAP HANA or Any-DB) and SAP BW/4HANA 2021, a shell conversion can be performed.

Shell Conversion: (T1) System Requirements

Shell Conversion: (T2) Pre-Checks

Shell Conversion: (T3) Custom Code Check

Shell Conversion: (T4) System Provisioning

Shell Conversion: (T5) Remote Conversion

Shell Conversion: (T6) Post Conversion Tasks

Shell Conversion: (T7) SAP Data Warehouse Cloud Core Tasks

Shell Conversion: (T8) Go-Live

Remote Conversion to SAP Data Warehouse Cloud

Remote Conversion: General Sequence

Remote Conversion (see figure 5) is offered by SAP to convert an SAP BW or SAP BW/4HANA system into SAP Data Warehouse Cloud. It enables customers to move whole data flows or transfer only selected data flows including data. Customers are able to decide whether they want to build a clean system, leave old and unused objects behind. For SAP BW systems on releases from 7.30 to 7.50 (running on SAP HANA or Any-DB) and for SAP BW/4HANA 2021, a remote conversion is planned. Attention: Currently not available, the Remote Conversion is planned for release with SAP BW bridge 2205.

Remote Conversion: (T1) System Requirements

Remote Conversion: (T2) Pre-Checks

Remote Conversion: (T3) Custom Code Check

Remote Conversion: (T4) System Provisioning

Remote Conversion: (T5) Remote Conversion

Remote Conversion: (T6) Post Conversion Tasks

Remote Conversion: (T7) SAP Data Warehouse Cloud Core Tasks

Remote Conversion: (T8) Go-Live

The post Conversion to SAP Data Warehouse Cloud: Conversion Paths and Cloud Transformation Steps appeared first on ERP Q&A.

One of the purposes of cloud software is to simplify, empower and enable all non-IT users with all about the benefits of corporate software investments, focusing on the “functional aspect” rather than all the challenges involved through new software implementation.

https://learning.sap-press.com/sap-and-the-cloud

SAP Data Warehouse Cloud continues using classic “measures and dimensions” concepts as the foundation for the highest level objects(“Model consumption” and “Perspectives”) also there exist some specialized roles in the data warehousing life cycle(data warehousing techniques, Non-relational DBMS, Multidimensional Models, ETL processes, connections, etc), that means that the technical layer (Data Builder, Data Integration, Connections, and Configuration) have to be done under best practices and using the most widely accepted techniques.

https://www.sap.com/insights/what-is-a-data-warehouse.html

Next, we are going to use a classic data warehousing approach to easily construct a basic data mart (“Consumption model” and “Perspective”) based on a classic star schema located under the Data Builder of SAP Data Warehouse Cloud since here is where all required data engineering gets performed.

Prerequisites:

1. You have a SAP Data Warehouse Cloud Tenant
   
2. You have your own Space
   
3. You have constructed a star schema(measures and dimensions associated with keys) located under Data Builder.

Creating Dimensions:

1. Log- On to your SAP Data Warehouse Cloud Tenant
   
2. Hit the Business Builder

Here is where you can define/reuse your dimensions from Data Builder:

Next, choose the dimension, it should exist deployed as View Dimension in the Data Builder layer, commonly time dimension is ever required in data warehousing(repeat steps for all dimensions needed):

If the dimension is correctly defined and under best practices for data warehousing, SAP Data Warehouse Cloud will detect its attributes and key definitions, it is very relevant since data relations of our multidimensional model(data mart) and every data warehouse using best practices should be defined by these key definitions. next confirm attributes and key definition auto-detection:

Finally, verify your attributes and key definitions, set your dimension as “Ready to Use” and save it:

repeat all previous steps for all dimensions needed

Creating measures (Analytic Set):

Click on New Analytical Dataset:

Next, select the corresponding Analytic Dataset containing measures and data relations to dimensions, it should exist deployed as Analytic Dataset(Business Entity) in the Data Builder layer

if the Dataset is correctly defined and under best practices for data warehousing, SAP Data Warehouse Cloud will detect its attributes, key definitions, and measures, it is very relevant since data relations of our multidimensional model (Data Mart) and every data warehouse should be defined by these key definitions. next confirm properties detected:

Finally, verify your attributes, measures, and key definitions, set your Dataset as “Ready to Use” and save it:

Next, we need to define data associations between measures and dimensions, click on “Associations” and click on add icon:

Select the required dimension and click on Apply:

Now click on Foreign Key Field, and select that one corresponding with the foreign key field located in the respective dimension, remember key relations is a common and widely used method for data warehousing construction:

Immediately an auto validation process is launched:

We will get 100% validation if data association integrity is correctly defined, it means that for every record in fac table(measures) exists at least one record in the dimension table, after that save it.

Creating Fact Model:

From here we will full define our classic star schema-based data mart, click on ”New Fact Mode”:

Define a name for your model and click on step 2:

Select your Analytic Dataset defined previously in the “Creating measures (Analytic Set)” section:

Next, a diagram with the associated objects will be displayed, check that there exist all the objects involved in your model, next proceed to include all the attributes that exist under the associated dimensions, click on add icon:

Select the corresponding dimension:

To continue click on step 3

Next click on “Link Association Path”:

next, the dimension view should be inside our fact view indicating that attribute for that dimension is now available for use, click on “create”:

Finally, verify that now our dimension is listed in the “Dimension Sources” section:

Continue clicking on “Attributes” and select all available attributes, repeat all previous steps for all other dimensions related to the fact table in order to complete all attributes:

when completed all dimension attributes association verify the final list:

Change the Status to “Ready to Use” and save it:

Finally, we need to expose the dimensions associated with our fact model when constructing the Consumption Model and Perspective, click on “Exposed Dimension Sources” and select the corresponding dimensions:

Repeat steps for all required exposed dimensions required:

Finally, check measures, attributes, and Exposed Dimension Sources sections to verify that everything is correctly defined

Creating Consumption Model:

At this point, we have constructed a basic data mart under a classic approach, however, there is missing security, and consumption best practices to fit most of the analytics solutions in the nowadays market, that’s the case of SAP Analytics Cloud and others, to do so, finally, we will construct a Consumption Model, it is very similar to Fact Model but with some relevant differences:

Click on “New Consumption Model”

Select base model fact, click on “on step 3” and “create”:

In the “General” section enable “Public Data Access”:

In the “Source Model” section, add and choose the dimension created previously under the “Creating Dimensions” sections:

Again as in previous sections, click on “Link Association Path” and repeat this step for all required dimensions:

Continue clicking on “Attributes” and “Measures” and select all available attributes/measures:

Creating Perspective:

Go to the “Perspective” section, define a significant name for your “Perspective” and select all available Measures and Attributes that also enable “Run in Analytical Mode” and click on “Deploy”:

As the last step, verify that all your constructed objects look similar to the next list image:

Finally, to validate the correct creation of all about our model, launch the Story Builder and verify there is listed our “Perspective” constructed all previous steps through.

The post Creating a basic Data Mart based on a classic star schema with SAP Data Warehouse Cloud appeared first on ERP Q&A.

Prerequisites

This blog post assumes that we’ve already set up a connection to a space within your Data Warehouse Cloud system.

Furthermore we’re going to need a table within the DWC Database User schema with a primary key defined. We’re going to walk this through below

Upserting

If the target database table has a primary key assigned, we can use UPSERTING to give us the behaviour we expect from the changed file. If an entry in our file has an entry with a matching primary key in the table, it will perform an UPDATE – if there’s no matching entry in the table it will INSERT the row instead.

For clarity, it’s important to note that any rows completely removed from the source file will not be removed from the target table.

Table Preparation

For our initial data load, we’re going to use a very simple schema. Copy the below contents and save it in a file called Employee.csv

1001, Adam Evans, 53
1002, Rose Hoskins, 32
1003, Fred Davis, 48

Within our Data Warehouse Cloud space, we want to navigate to the Database Access section. Here we find the Database User that was specified in our Data Intelligence Connection. Select this user on the left then click on Open Database Explorer

Once we reach the Database Explorer, we see a Database Connection Properties popup box. Here we want to enter the password for our Database User, and optionally save these credentials in the SAP HANA secure store before we click OK.

For the purposes of this blog post we’re going to directly create the table using the SQL Console. In a productive scenario you would probably want to define this table in a way that can be replicated and version controlled (perhaps using .hdbmigrationtable)

To open the SQL Console, we right click on the connection we just created, and then select Open SQL Console

We want to use the below SQL Syntax inside the SQL Console to create our target table, specifying the Employee ID (empID) as the Primary Key

CREATE TABLE EMPLOYEE(
	empID varchar(255) NOT NULL PRIMARY KEY,
	empName varchar(255) NOT NULL,
	empAge int NOT NULL
);

Once the command has been copied into our console, we run it to create our target table

Once our table has been successfully created, we’ll see a message with some execution statistics. Additionally, if we want to verify it for ourselves we’ll find the table in the catalog on the left

Creating Our Pipeline

First things first, we’ll want to upload our Employee.csv file to one of our Data Intelligence Connections using the Metadata Explorer. For this blog post we’re using a folder within DI_DATA_LAKE, but this choice is arbitrary

Within the Upload screen, click on Upload in the top right, select your Employee.csv file from your computer and then click on the Upload button on the bottom right

Once our file has uploaded, we can head to the Data Intelligence Modeler. Create a new Pipeline (Graph), then add and connect the following three Operators in sequence: Structured File Consumer->Table Producer->Graph Terminator

The Structured File Consumer will take our csv file from the DI_DATA_LAKE and read it in a form that can be processed. The Table Producer will then take this data and use it to UPSERT records into the table we just created. After this has completed, the Graph Terminator will end the processing of our Pipeline

Let’s get our operators set up. First, select the Structured File Consumer, then edit the Source Object

Select the SDL (Semantic Data Lake) Service, Connection ID (DI_DATA_LAKE) and the Source (File Path to our Employee.csv file) then click back

Next, we’re going to set up our Table Producer Operator

Set the Source to HANA_DB if it isn’t already set. Select our Connection ID (DWC_UPSERT), then click on the Target Box

Select the Open SQL Schema that was created in Data Warehouse Cloud when creating the Database User (in this example, UPSERTDEMO#DIUSER)

Select our Employee table, then click OK. Select the Upsert checkbox, and map our three table columns to C0, C1 & C2 in our Target Columns. Select the back arrow

Save our new pipeline. Before we run our Pipeline, we’ll need to employ a quick workaround. When creating our Database User in DWC, the Schema is created with the hashtag character (#). However, in Data Intelligence the Table Consumer doesn’t allow the hashtag character in its targets.

Switch back to the Diagram view, then Save and Run the Pipeline

Success

Once our Pipeline Run has been successfully completed, we’ll see it at the bottom of the Modeler

We can also verify the contents using the Data Intelligence Metadata Explorer

The post SAP Data Intelligence Cloud – File Upsert into SAP Data Warehouse Cloud appeared first on ERP Q&A.

This blog post is about a strategic feature of SAP Data Warehouse Cloud, namely the Data Marketplace. It helps data providers and data consumers to exchange data in clicks, not projects – to heavily reduce the data integration efforts which currently costs Time, Budget & Motivation in analytics projects. It consequently addresses use cases ranging from external data integration and harmonization to cross-company data collaboration between business partners that use SAP Data Warehouse Cloud. For the time being, commercial transactions remain outside of the data marketplace and an embedded license management allows a BYOL (bring your own license) scenario to onboard existing Data Provider-Data Consumer relationships or new ones done on the SAP Store or via the Data Provider’s sales channel.

This blog provides an overview of the data marketplace as well as shows the core steps of the processes for data consumers and data providers with links to detailed blogs.

Timing Information: The Data Sharing Cockpit of the Data Marketplace to list Data Products is already productive with wave 2022.1 on selected SAP Data Warehouse Cloud tenants. With Wave 2022.2 – delivered in CW3 in 2022 – the Data Marketplace will be enabled in all SAP Data Warehouse Cloud tenants. In this blog post, certain topics that are foreseen but not yet communicated on the roadmap are stated with “in the product vision”.

1. Data Marketplace Value Proposition

Data Consumer – Access in Clicks, Not Projects.

From the realization of outside-in use cases to the creation of a holistic data platform/warehouse, the necessity to include external data is more important than ever. What do we mean by external data? Let us give you some examples & use cases :

• COVID-19 numbers that an HR department needs to understand the impact for their office locations
  
• Stock Data of competitors to put internal performance data into a market perspective
  
• Company Master Data & KPIs to assess how much of the market is already assessed or to measure market attractiveness in the area of market expansion
  
• ESG data to understand the risk & opportunities arising from the sustainability setup of your ecosystem
  Internal data from your business partners that want to share the data with you as part of a business process, e.g. point-of-sale data for promotion evaluation

In a nutshell, with external data, we mean all data that you cannot extract from your company’s internal applications but get access to from an external party which can be a commercial data provider, an open data source, or a company where data is traded as part of a business process.

While this data is super helpful – especially in times of market disruption through innovation, pandemic times, or economic climate changes where historical data does not help you to project the future – integrating it can be super cumbersome and is today often the hurdle for data-driven innovation and cross-company collaboration. Why can this be the case?

• Flat Files Chaos: External Data is available in form of flat files where ingestion is cumbersome and for ongoing updates impracticable, ungoverned and expensive
  
• API Skills: External data is often provided as an API where skills in the business department is lacking and the integration requires an expensive project and support from IT
  
• Export Struggles: External Data is available in a dashboard where it can be exported where in the import processes challenges such as data type definitions create a headache
  
• Real-Time Setup: External Data is available as a real-time query where only selected elements are required and technical means to provide that are lacking
  
• No Experience: External data – especially from business partners – needs to be exchanged and no platform exists with which the data can be exchanged in a scalable fashion – most often solved with mail attachment exchange

In a nutshell, for enterprise-grade integration of external data with the current setup, the integration work lies with the data consumer, and almost always an IT project is required which in reality leads to the fact that the use case is not realized and potentials remain unaddressed.

With the Data Marketplace, this changes completely for the data consumer in three steps:

CONSUME. The data consumer can access external data in a few clicks into his SAP Data Warehouse Cloud without doing any integration work. All of the integration work is done by the Data Marketplace (after the Data Provider has once onboarded his views). This holds true for the initial load as well as for updates published by the data provider. As a consequence, the Data Marketplace allows you to standardize the inbound flow of external data where in a company today a mess of different setups makes it impossible to scale.

COMBINE. Once the data is integrated via the Data Marketplace, you benefit from the entire data management functionality of SAP Data Warehouse Cloud to cleanse, harmonize and prepare the data for consumption. One explicit feature to call out here is the Intelligent Lookup that helps you to bring together datasets that do not have a singular, technical JOIN condition which is almost always the case with external data. And ultimately, you can then easily consume the data with SAP tools such as SAP Analytics Cloud, SAP HANA Cloud, SAP Data Intelligence, or 3rd Party Tools such as Jupyter Notebook, PowerBI, etc.

CONTROL. Ultimately – by being tightly embedded into the SAP Data Warehouse Cloud platform – the Data Marketplace allows you to setup an infrastructure with which you can manage your data inbound processes at scale, e.g. by leveraging spaces. Within the Data Marketplace, you benefit from Access control mechanisms that manage who can access which Data Product. This way a provider can represent his existing contracts as a digital twin and onboard his existing customer relationships while new ones can be created via the SAP Store.

Data Provider – Easily deploy data for SAP customers

In order to realize the above-described data consumer perspective, the data providers obviously play a crucial role. But why should they care?

1. User Visibility. With the growth of potentially relevant data sources, the actual users of a data solution often do not even know about potential data offerings in the market. As a data provider, you need to be visible to the end-users that are pivotal in the decision of which data to acquire and how.
   
2. Share of Customer Wallet. Data Providers that have contracts with their clients always need to find an individual solution to make the data accessible within an SAP Data & Analytics infrastructure. The required budgets that are reserved for data integration efforts – in-house or by consultants – endanger a potential deal or decrease the potential size of the data deal.
   
3. Adoption Risk. More and more data products are sold in a subscription or pay-per-use model. Consequently, a complex integration & usage experience endangers the adoption and customer relationship.
   
4. Customer Journey Friction. With an inefficient data delivery mechanism, it is difficult to position and deliver additional offerings. This way it is difficult to land and expand while at the same time most of the trust is built on small scenarios that prove their value and are then expanded into other units.
   
5. Commercial Friction: With several players involved in the external data value chain, as a data provider, you need to choose your battles wisely.

How does the Data Marketplace for SAP Data Warehouse Cloud help?

1. Visibility into SAP customer base: The Data Marketplace is provided as a free in-app solution within SAP’s strategic public cloud data platform – SAP Data Warehouse Cloud. With the Listing in the Data Marketplace, you reach SAP’s customers that look for promising data products that help them solve their problems.
   
2. Standardized Deployment Option: With the Data Marketplace, there is a clear deployment option where the provider benefits from standardized processes that establish the same experience across data providers for the customer.
   
3. Adoption Insurance: With the Data Marketplace, the provider’s customer gets access to the data in a few clicks with the right semantics in place. This ensures a great first experience as well as ongoing usage through a process that realizes seamless consumption of data updates (without an IT project required).
   
4. Marketing-Mix Modeling & Delivery: With the listing process in the data sharing cockpit, you can efficiently design Data Products that align with your GTM strategy while keeping the operating costs minimal. This way you can start the individual customer journey with a small listing that you can extend over time – all configurable via the UI.
   
5. Flexible Data Product Licensing Options: On the one hand side, you can bring and represent your own licenses in the License Management which means there is no brokerage fee taken by SAP for executing data exchange via the Data Marketplace. On the other hand, to support a data provider in the commercial process, options to list on the SAP Store or via the Partner Datarade will be offered.

All further information on the Data Provider perspective shall be answered in the E2E processes below or in the Provider FAQ on the SAP Help Page right here.

2. Data Marketplace End-to-End Demo

The best way to understand the value proposition of the Data Marketplace is to see it in action.
Feel free to take a look at the following 25-minutes long end-to-end demo video where you see how 2 data products are loaded and an additional one is listed and delivered. In addition, the Intelligent Lookup is shown to demonstrate how the native SAP Data Warehouse Cloud functionality helps you to work with the acquired data in an SAP context.

3. Data Marketplace Setup Overview

In order to truly understand the benefits of the Data Marketplace – as a data consumer or data provider, it is important to know how it operates. This chapter should give an overview and understanding of the setup while the following chapter will give more context for the processes.

First of all, all participants in the Data Marketplace need access to an SAP Data Warehouse Cloud tenant. For companies that only want to use SAP Data Warehouse Cloud to offer and deliver data in the Data Marketplace and do not want to use it internally as a data warehouse solution.

As a Data Provider, you use the Data Sharing Cockpit to list one or multiple SAP Data Warehouse Cloud artifacts as a so-called Data Product. For this Data Product, you can manage access via the License Management and updates via the Publishing Management. Based on this definition, a hidden “Data Product Space” is created in the Data Provider’s SAP Data Warehouse Cloud Tenant.

When a Data Consumer now discovers such a Data Product that he wants to load, he needs a license key to activate it (unless it is a free product) and can select the target SAP Data Warehouse Cloud Space in which he wants to consume the Data Product. Based on this selection, the Data Marketplace now creates a database connection between the generated “Data Product Space” and the selected target space.

Subsequently, the defined artifacts are automatically created and the data is replicated. Every time a new update is made available to the data, a new replication is triggered. Federated access is currently being investigated while replication was the priority to mirror the current process of flat-file delivery where a physical copy is shipped as part of the data contract.

This approach scales especially if multiple customers consume the same product (or products based on the same data) as the Data Provider only once needs to connect, list, publish and update the product while all data consumers that have subscribed to the product benefit immediately (or manually if they choose to manually control the update flow).

Consequently, the Data Marketplace connects all SAP Data Warehouse Cloud customers (currently within the same landscape, e.g. EU10, US10, etc.) in a matter of clicks, without an IT project required.

4. End-to-End Data Marketplace Process

In this chapter, you get a high-level walkthrough of the main data marketplace processes.
Further step-by-step guidance and field-level explanations can be found in more detailed blogs then.

Data Provider – Connect & Prepare Data for Listing

In order to make data available on the Data Marketplace, you need a deployed graphical or SQL view in your SAP Data Warehouse Cloud populated with the data that you want to ship. Based on the origin and complexity of your data product, different onboarding mechanisms are possible. In a nutshell, you can distinguish between the following 5 archetypes:

1. The easiest while the most manual thing is that you can create a local table inside SAP Data Warehouse Cloud and use the Data Editor to enter data via the UI. This might be useful for small helper tables or if you need a certain party to submit a few entries in an integrated fashion, e.g. enter reference that for which data shall be delivered.
   
2. Upload local CSV files via the Desktop Upload functionality of SAP Data Warehouse Cloud. Once the Table is created and initially published, you can also add new data – incrementally or as a full load. This is a simple setup but also requires manual effort to realize.
   
3. The third option might be the most common one – connecting a cloud storage provider such as AWS, Google Cloud, or Microsoft Azure and loading files (CSV, XLSX, PARQUET, etc.) via the Data Flow functionality of SAP Data Warehouse Cloud. This is especially handy as you can schedule data flows and conduct ETL processes such as cleansing the data or adjusting data types.
   
4. The fourth option is to connect a database to SAP Data Warehouse Cloud. This can be interesting as this connection type allows to use a federated setup. This means that the data is not replicated to the Data Provider DWC tenant but live access is established. Consequently, it is the provider’s responsibility that the setup is robust in order to ensure that in case a data load is triggered via the Data Marketplace, the source system can be called. That is why snapshotting of the view is possible on view level.
   
5. Last but not least, you can create a so-called OPEN SQL schema or Database user in the Space Management of SAP Data Warehouse Cloud which provides you with the credentials to connect via the HANA DB Client. This allows you to data write with 3rd Party Tools into SAP Data Warehouse Cloud and then create views on the table.

In a nutshell, multifold data onboarding options are available and you can find a full list of supported sources systems right here.

1. Data Provider – Create Data Provider Profile

Once the views are created that you want to list in the Data Marketplace, the processes in the Data Sharing Cockpit start. You need to create a Data Provider Profile or assign your user to an existing one via an activation key. With the Data Provider profile, you can describe your area of expertise, connect your LinkedIn page and tag the Industries, Data Categories, Regions that you serve with your data. In addition, you maintain the contact details to help curious data consumers to reach out.

2. Data Provider – Create Data Product Listing

Data Product Listing is the process of defining the marketing mix of your data offering. For interested data consumers, you can provide all “4Ps” as you would call it in economics: Product, Price, Promotion, Placement.

It starts with descriptive information. You can maintain a free text that describes the product, its use case, its sources, etc. With the same taxonomy as in the Data Provider profile, you can tag your product with the applicable Data Category, SAP Application, Industry, and Country to optimize the search experience and likelihood to be found. In addition, you can maintain images as well as data documentation such as a metadata catalog or KPI definition document.

A further very important asset is the Sample Data representation. This is currently been achieved with a JSON upload () while later this will be realized with a filter on the actual data set. One or multiple samples can be maintained – ideally one for each artifact that is part of the data product – with each maximum 1000 records.

A further key setting is the maintenance of the delivery information. You can decide between the following data shipment types:

• Direct Delivery: Data is shipped via the Data Marketplace into the space that is selected in the Load Dialogue. The prerequisite is that the Provider has connected his assets to his Data Provider tenant. There are then two delivery pattern options:
  • One-Time: Nature of the data doesn’t require data updates/refresh of data, thus no publishing management is required.
  • Full: There are data updates expected/planned that can be managed with the Publishing Management. Each delivery represents a full load of the existing
• Open SQL Delivery: The provider is capable to push the data into an OpenSQL Schema that the Data Consumer creates. Consequently, the Data Provider does not need to onboard his data to a DWC Tenant prior. This makes sense especially for highly tailored products or validates demand before investing in the Direct Delivery setup.
• External Delivery: The data product is listed in the Data Marketplace but the delivery is taken care outside – e.g. flat-file delivery, cloud storage access, dashboard access. In this setup, it is the full provider’s responsibility to deliver the data to each individual consumer and each consumer’s responsibility to integrate the data themselves.

Furthermore, commercial information can be maintained. In general, the Data Marketplace runs on a “Bring your own License” setup. You can represent your existing license in the Data Marketplace to authorize access. This can be a license that a provider sells in his own sales channels or via SAP channels, the SAP Store. In addition, you can set your product to “Free of Charge” which consequently does not require a license key to access a product. For license products, you can then maintain price information as well as terms and conditions.

Last but not least and most importantly, you select the artifacts that you want to ship in an automated fashion (in case you have a direct delivery data product). In the most simplistic fashion, it can be one, entire view that you can ship as it is. At the same time, with one data product, you can also ship multiple views at once, e.g. transactional and master data views that can be joined on the consumer side. Furthermore, you can use data filter or column selection to base multiple data products on the same base view to accommodate the portfolio and GTM strategy.

Once you have defined these Data Product listings aspects, you are ready to list your product on the data marketplace. The entire process can be done in several minutes in case the Data Provider knows what exactly he wants to list and deliver.

The next two steps are only required for specific Data Product setups:

• 3. Creating Releases in the Publishing Management is required for Data products with Delivery Type “Full”. Only with a defined data release, you can set the Data Product status to “Listed”.
• 4. Creating Licenses is only required for commercial data products where access shall only be granted to users that have a contractual right to do so. This can either be for commercial products that customers have to pay for or for a free of charge product that should only by accessible with explicit permission of the data provider via license.

3. Data Provider – Create Releases

Most Data Products are being updated frequently or infrequently. This is where non-integrated setups create efforts for both data providers and data consumers. Data Providers need to create new data slices and data consumers need to find ways to ingest new incoming data. To reduce the friction of releasing and consuming data updates was one of the biggest motivations to build the Data Marketplace. This is why a dedicated module is available in the Data Sharing Cockpit with the Publishing Management.

For each Data Product, the Data Provider can easily define a new release that provides the Data Marketplace with the information the new content is available. Data Consumers that have activated the Data provide immediately receive the new data into their existing artifacts without any intervention required. At the same time, a “Manual Update” mode is available as well on the consumer side that allows the data consumer to decide when the new data shall be ingested. The Data Provider has also the option to lock a data product for the time where he updates his view to make sure no inconsistent data is shipped (e.g. to prevent data from being shipped to the customer while it is updated by the data provider).

In the first release of the Data Marketplace, a new data release always provides the full data set in the provider’s view(s) to be transferred. In the product vision, you will be able to define and ship incremental releases as well.

4. Data Provider – Create Licenses

Finally, the Data Provider is able to set up licenses that make the commercial setup actionable. Therefore the Licensing Management module exists in the Data Sharing Cockpit. A license entitles one or multiple users to access one or multiple data products for a limited or unlimited time. In addition, a domain check can be set up to only allow users with a certain mail address, e.g.@sap.com, to activate a specific license. For a given license, the data provider can generate as many activation keys as he likes and can send them to the data consumer that can enter the license on the specific product page or in the “My Licenses” section.

There are four major licensing models that you can execute with the License Management:

• Product Licenses: You set up a license that only contains one product and has no customer-specific settings. Each customer that buys the product license gets an activation key and can use the product.
  
• Product Group License: You set up a license that contains multiple data products that are licensed together and the license also has no customer-specific settings. Each customer that buys the product group license gets access to all products that are maintained in the license. The Data Provider can add new Data Products to the license at any time.
  
• Contract License: Compared to the Product and Product Group License, a contract license is always specific to a specific customer contract. You can maintain e.g. the Contract ID in the reference as well as the contract dates in the validity. All products that are part of the contract are maintained in the license and only the users of the specific customer get an activation key.
  
• Trial License: As you can restrict the validity and product scope, you can use the License Management to set up a trial license that gives access to a product for a limited time or to selected products of a product group. In the product vision, you will be able to set a certain trial option in the data product and the data consumer will be able to sign-up for trial access in a self-service manner while the system generates the trial license in the background.

As the Data Products and the License Management is decoupled, any combination is possible. In the product vision, you will also be able to set a data filter within the license to restrict data access via the license to a certain data scope.

The Data Consumer part is under construction and will be released with the General Availability of the Data Marketplace in CW3/2022.

<Data Consumer – 1. Find Data Products >

<Data Consumer – 2. Elaborate Data Products>

<Data Consumer – 3. Load Data Products>

<Data Consumer – 4. Track Data Deliveries

<Data Consumer – 5. Use Data Product’s artifacts>

5. Data Marketplace Use Cases

Many that are new to the topic of external data and data sharing are curious about the use cases that can be covered with this cross-tenant data exchange technology. In the following, I would like to give an overview of the archetypes of use cases that are possible. Check the chapter below on the Data Provider and Data Product Portfolio below.

1. Outside-in Analytics: Most SAP Data Warehouse Cloud customers focus on internal (SAP) data. With the Data Marketplace, you can start to add an outside-in perspective, e.g. by adding Benchmark Data or Market Data to your SAP S/4HANA Sales & Distribution Data.
   
2. Holistic Planning & ML/AI: In current dynamic times, it gets more and more difficult to plan and predict your business outcomes solely by extrapolating internal data. This is where the Data Marketplace helps with external data such as macroeconomic and microeconomic data as well as environmental and geospatial data. Concrete Examples could be to assess future markets based on GDP, assess future suppliers based on ESG risk factors or assess future locations of e-charging stations based on movement data.
   
3. Cross-Company Data Collaboration: The Data Marketplace and the Data Sharing Cockpit is not only for commercial and public datasets but can be used for B2B data exchange as well. Anyone can list Data Products to exchange data with other SAP Data Warehouse Cloud users. Due to the tight connectivity with SAP source systems this is especially interesting when the participants want to exchange SAP data. This way e.g. retailers can share data with manufacturers or manufacturers with suppliers or construction companies with the many companies they collaborate with when the many stakeholders in a building project.
   
4. Internal Data Marketplace: Interestingly, in large multinational companies the same base requirements for data sharing hold true as in cross-company data sharing. If you share data with another unit or subsidary, you want to demonstrate the data product transparently and manage the governed access. Consequently, the functionality of the Data Marketplace can be used when exchanging data internally as well. While in external data cases contracts and payment flows are the basis for license management, it is then internal orders and department missions.
   
5. Faster Data Modelling with Data Crafting Assets: This last use case is rather specially and mentioned has it was a learning while using SAP Data Warehouse Cloud in concrete use cases. Often times, you miss those little helper tables that make your modelling life easier, e.g. when the one view has the country identifier with two-digit ISO codes and the other one the country name, or when the date information does not have the same granularity, etc. This is where SAP lists those data crafting assets that help users in data modelling challenges and we cannot see other Consulting Companies to share their crafting assets as well.

6. Data Provider & Data Product Portfolio

<under construction for GA in CW3/2022>

7. Roadmap

<under construction for GA in CW3/2022>

1. Data Product Lifecycle Management
2. Live Access Data Products
3. Data Provider Shop Integration
4. Visibility Management
5. Public APIs

The post SAP Data Warehouse Cloud, Data Marketplace: An Overview appeared first on ERP Q&A.

Step 1: Import a Dataset with Data Builder

One way to bring data into SAP Data Warehouse Cloud is by importing flat files. A flat file is a file that does not contain links to other files or is a non-relational database. It stands on its own, for instance, like a single text-only file. The most common flat files are .txt and .csv files.

To access the Data Builder, simply click on the Data Builder icon on the left-hand side menu of SAP Data Warehouse Cloud.

Simply click on the import CSV icon as shown in the image below.

Quickly review your data and then click deploy once done.

Add in your remaining CSV files by repeating the process.

There are also other ways to bring in your data into your system. SAP Data Warehouse Cloud supports a wide range of connections to data sources. These can be connected in the Connections tab of your Space properties page. These include:

• Adverity
• Amazon Athena
• Amazon Redshift
• Amazon Simple Storage Service
• Cloud Data Integration
• Generic JDBC
• Generic OData
• Google BigQuery
• Google Cloud Storage
• Hadoop Distributed File System
• Microsoft Azure Data Lake Store Gen1
• Microsoft Azure Data Lake Store Gen2
• Microsoft Azure SQL Database
• Microsoft SQL Server
• Oracle
• SAP ABAP
• SAP BW
• SAP BW/4HANA Model Transfer
• SAP ECC
• SAP Fieldglass
• SAP HANA
• SAP Marketing Cloud
• SAP S/4HANA Cloud
• SAP S/4HANA On-Premise
• SAP SuccessFactors for Analytical Dashboards

Step 2: Create a Graphical View with Data Builder

In SAP Data Warehouse Cloud, you can use the graphical view builder to easily create data views. This allows you to work intuitively without having to be familiar with SQL statements.

In the graphical view builder, you have many resources to model your data, combine them from many sources and assigning business semantics that make your output easier to understand. Here is an example of a graphical view modelled in SAP Data Warehouse Cloud.

Go to the Data Builder and click on the New Graphical View button.

To start building your model, click and drag the SalesOrders table onto the canvas.

As you can see, an output node appears on the canvas as soon as your drop your table in it. The output node is where all of our join table information will appear once you’ve completed the model.

Click on the output node and then click on the data preview button to see a preview of the sales orders data.

Next, drag the table SalesOrderItems on top of the SalesOrders table to join the two tables. The icon that has appeared is our join node called Join One. The column SalesOrderID from both tables is automatically joined based on entity-relationship model.

Expose for consumption from SAC

Save and deploy

Step 3: Create a SQL View with Data Builder

In SAP Data Warehouse Cloud, you can also use the SQL view builder to create data views with SQL statements.

Drag the Table to the editor

Add SQL statement

Validate the SQL:

SELECT A.* FROM "BusinessPartners" A
INNER JOIN "Addresses" B
ON A."ADDRESSID"=B."ADDRESSID"

All the attributes would pop up.

Expose and deploy it.

Step 4: Create a Dimension with Business Builder

In SAP Data Warehouse Cloud, click on the Business Builder icon.

Then, click on New Dimension.

Select the data entity you wish to use in this dataset. Either click on the entity you want to use, or search for it by using the search field on top of the pop-up window.

Change the Business Name and Save it.

Now it’s time to add attributes to your dimension.

Click on Attributes on the top menu. To add existing attributes, click on the Add Source Attributes button.

Add a key definitions

Step 5: Create an Analytical Dataset Associated with Dimension with Business Builder

The Business Builder helps you instantly model your organizational data in business terms. With it, you can create business entities and relationships with an easy to understand structure, making business users more independent from IT.

In practice, it allows business users to collaborate and create data models in SAP Data Warehouse Cloud based on a semantic view of the organization’s data. Once all the data is setup and made available by IT in SAP Data Warehouse Cloud, then it’s easy to give permissions to the right people, enabling you to get your data ready to provide business insights.

To create your business scenarios with the Business Builder, here are the objects you can create:

– Business Entities (Analytical Datasets or Dimensions)

– Fact Models

– Consumption Models and Perspectives

– Authorization Scenarios

Business Entities: Analytical Datasets or Dimensions

A business entity is based on a view created in the data layer, and it represents an object, such as information about products or sales, including all selected attributes and measures. In the Business Builder, you can create associations between facts and attributes coming from a view in the data layer. Business entities can be either an analytical dataset – which is a fact table with measures and foreign keys – or a dimension – which is tables containing only attributes.

Business entities are independent from the data layer and include versioning to allow you to have a high level of control.

Fact Models

A fact model is a combination of multiple business entities, including facts and dimensions, as well as added calculations across multiple facts. Typically, fact models are used to show more complex yet generic metrics or KPI’s, such as Pipeline Coverage. Fact models can be nested, which can help make this model very re-usable.

Consumption Models and Perspectives

Perspectives allow you to select only specific attributes and measures and focus on them to create the best input to visualization and consumption tools, like SAP Analytics Cloud.

A consumption model allows you to combine business entities and fact models to create an output that answers a concrete business question. A consumption model can contain multiple perspectives. You can enrich business entities with additional calculations if necessary. Once you are done with a consumption model, you can deploy it to any target visualization client, such as SAP Analytics Cloud or 3rd-party tools via SQL.

To model a meaningful consumption model, business entities define associations between each other. All potential association targets can be pre-defined on the data layer to provide business users with a variety of modeling options when preparing their use case-specific consumption model. Designating the right measures to your business entities is the first step towards fully utilizing the powerful capabilities of the Business Builder in SAP Data Warehouse Cloud.

Authorization Scenarios

An authorization scenario is a way to restrict access to parts of the data within a business entity. It’s reusable and can be applied to any business entity in SAP Data Warehouse Cloud.

In SAP Data Warehouse Cloud, click on the Business Builder icon. Then, click on New Analytical Dataset.

Select the data entity you wish to use in this dataset. Either click on the entity you want to use, or search for it by using the search field on top of the pop-up window.

Click on Create. Change Business Name and Save it.

Now that your analytical dataset is created, it’s time to populate it. Start with adding measures.

Click on Measures on the top menu. To add existing measures, click on the Add Source Measures button.

A pop-up will open to show you the measures available in the source data. Check all the measures you wish to include. Then click on Apply.

Now it’s time to add attributes to your analytical dataset.

Click on Attributes on the top menu. To add existing attributes, click on the Add Source Attributes button.

A pop-up will open to show you the attributes available in the source data. Check all the attributes you wish to include. Then click on Apply.

Add an association with a Business Entity:

Add the Dimension that we created before as the association:

Select the target key mapping and Save it.

You are almost done creating a base analytical dataset. Before you move on to another task, it’s important to preview your data and make sure you have the right dataset.Just click on the Data Preview link on the top right-hand corner of the screen.

Step 6: Create a Consumption Model with Business Builder

If we’re using the Business Builder, we will need to create a Consumption Model and Perspective to be consumed in SAC.

Consumption models in the Business Builder of SAP Data Warehouse Cloud are the final business model you need to create before sharing your data with a visualization tool, such as SAP Analytics Cloud or equivalent.

Click on the New Consumption Model button.

Enter a name for your consumption model.

Click on Step 2.

Select the Initial Fact Source. This can be an analytical dataset, a dimension, or a fact model.

Click on Step 3.

Define an alias name for the source in Source alias.

Click on Create.

Add Source Context:

Add Measures and Attributes:

As mentioned before, you need to make your consumption model available to visualization and reporting tools, such as SAP Analytics Cloud, before these tools can actually have access to the output of this model. This is really simple to do, and you can grant public data access at any time, as well as revoke it again.

Add Perspectives:

Save and deploy the perspectives:

Step 7: Expose from SAP Data Warehouse Cloud

In order to ensure the data entities and models created in your SAP Data Warehouse Cloud tenant are consumable by SAP Analytics Cloud, it is important to expose them for consumption.

• If the entity was created in the data builder, simply click on the entity and set the Expose for Consumption toggle to ON as below.
  • Expose for Consumption: ON
• If we’re using the Business Builder, we will need to create a Consumption Model and Perspective to be consumed in SAC.
  • Allow public data access: checked
  • Consumption Model
  • Perspective

If the entity was created in the business builder, the toggle shows up as a Allow public data access checkbox in it’s properties tab. Simple click on the data entity to open it, and ensure the checkbox is ticked.

Step 8: Consume in SAP Analytics Cloud

To link the SAP Analytics Cloud to the SAP Data Warehouse Cloud tenant, navigate to Configuration on the bottom left, and select Tenant Links.

When you link your tenants, you’ll enable the product switch in the top right of the shell bar, and be able to easily navigate between them.

Go to SAP Analytics Cloud trial, click on connections then click on the + icon to create a live data connection setting. Select connect to Live data then choose SAP Data Warehouse Cloud.

Provide a name and hostname

From the home screen of SAC, select stories and click canvas type of stories under create new.

Click on Add data button to add data source with your DWC:

Again, click on Add data button to add data from existing data model of your DWC:

This will bring you to the data analysis section where you can analyze the data from the model.

Switch to the Story Tab:

Draw your chart as per your business requirement.

The post SAP Data Warehouse Cloud: End-to-End Data & Analytics Scenarios appeared first on ERP Q&A.

This blog post is about a strategic feature of SAP Data Warehouse Cloud, namely the SAP Data Warehouse Cloud, SAP BW bridge. SAP offers its customers through RISE with SAP, the opportunity to move with a business transformation as a service (BTaaS) to the cloud. For BW customers, this means SAP BW to SAP Data Warehouse Cloud. SAP positions SAP Data Warehouse Cloud as the strategic target solution for Data Warehousing in the public cloud, with SAP BW/4HANA Private Cloud Edition (PCE) as an option to start the transition. In this context, SAP BW bridge offers customers the opportunity to implement their new data warehousing use cases directly in the cloud environment while protecting, and retaining their existing on-premises investments. The blog provides an overview about SAP BW bridge, explains how to move from an existing SAP BW system to the cloud, and gives insights with a complete end-to-end greenfield scenario including a system demo in SAP Data Warehouse Cloud.

Overview

SAP Data Warehouse Cloud is SAP’s offering for all data warehousing use cases. This SaaS (Software-as-a-Service) is based on SAP HANA Cloud. It combines data, and analytics in a cloud solution that offers data integration, database, data warehouse, and analytics services. This enables customers to realize the full potential of a data-driven business. As a way to improve the integration with SAP ERP systems, the SAP BW bridge enables ABAP-based data extraction and staging capabilities within SAP Data Warehouse Cloud (see figure 1).

In the future, a tool-based transfer of existing SAP BW and SAP BW/4HANA staging scenarios will be enabled. Then the SAP BW bridge will enable a seamless transfer of existing ETL processes in a dedicated SAP BW bridge Space in SAP Data Warehouse Cloud. Here, the extensive functions of ODP extractors, and ABAP code within the SAP Business Technology Platform (SAP BTP) ABAP environment can be adopted in SAP Data Warehouse Cloud using the Cross-Space-Sharing approach.

SAP BW to SAP Data Warehouse Cloud

Within SAP BW bridge, customers are able to implement data extraction and staging scenarios up to the CompositeProvider level. In other words, it is not possible to create new queries within the SAP BW bridge environment. In this regard, within the SAP BW bridge, there is no support for OLAP engine, and functionality dependent on OLAP (e.g., analysis authorizations, query as info provider, query execution). Front-End tools do not have the possibility to access SAP BW bridge artefacts directly.

The SAP BW bridge environment is primarily intended for ODP-based source systems, which means that the connection scenarios only become available via Operational Data Provisioning (ODP). Non-SAP sources will be connected directly to SAP Data Warehouse Cloud (see figure 2). Objects from source SAP BW system(s) will be converted to the SAP BW bridge environment using Conversion Tools, including the SAP BW queries.

To take the full advantage of SAP’s data warehousing offerings, customers today need to deploy both SAP BW/4HANA, and SAP Data Warehouse Cloud. In the future, the SAP BW bridge will enable customers to merge these offerings into a single data warehouse solution in the cloud. With SAP BW bridge, SAP addresses BW customers that are looking for a way forward from SAP BW NetWeaver, and SAP BW/4HANA (See figure 3).

SAP BW Customers will have the option in 2022 to convert their existing on-premises investments to the cloud via remote, and shell conversion. First, the SAP BW bridge conversion will be offered for SAP BW 7.4, and SAP BW 7.5 systems (initially as shell conversion, followed as remote conversion), subsequently the conversion for SAP BW 7.3 systems (shell and remote conversion) will be available. Additionally, the conversion will be available for SAP BW/4HANA 2021 (shell and remote conversion). Regarding release coverage, please consider the details in the roadmap. Customers with lower SAP BW releases will need to upgrade their system(s) first, and then convert the required scope to SAP BW bridge in SAP Data Warehouse Cloud. Please note that SAP BW systems 7.40, and lower are already out of maintenance (see figure 4).

The SAP BW bridge artefacts out of the Business Technology Platform (SAP BTP) ABAP environment are available via remote tables using SAP HANA Cloud Smart-Data-Access (SDA) in a dedicated SAP BW bridge Space in SAP Data Warehouse Cloud. The remote tables in the SAP BW bridge Space can then be used in the regular SAP Data Warehouse Cloud Spaces via the SAP Data Warehouse Cloud cross-space sharing approach.

Add-ons are not supported in SAP BW bridge. Therefore, planning is not available within SAP BW bridge. In this regard, SAP positions SAP Analytics Cloud Planning as the planning application, and in the future (earliest end of 2022) SAP Data Warehouse Cloud as the planning foundation for the data. Application development is not supported in SAP BW bridge. Any app development should be done via the Business Technology Platform App Building on SAP HANA, for which customers need to license, and use the stand-alone version of SAP Business Technology Platform ABAP Environment.

Target Scenarios for SAP BW bridge

Greenfield with SAP Legacy Sources

Customers building a new data warehouse in the cloud with SAP Legacy systems as data sources that will only be migrated to a cloud-based system in the future. Expecting the same level of data integration, and convenience functions as known from SAP BW/4HANA.

Conversion with SAP BW NetWeaver & SAP BW/4HANA

Customers with an SAP BW (SAP BW 7.3 and upwards, any DB) moving their data warehouse to the cloud expecting to retain their data, and their legacy data flows, but renovating their data consumption layer with SAP Analytics Cloud or 3rd party clients on top, and expanding their data footprint to cloud, and non-SAP sources.

Hybrid with SAP BW/4HANA

Customers with an on-premise SAP BW/4HANA looking for a path into the cloud for their data warehouse workload. Starting with hybrid scenarios for consumption to combine SAP BW/4HANA data, and SAP Data Warehouse Cloud data, and then easily moving more and more of the SAP BW/4HANA data flows to the cloud, and successively transition them to modern SAP Data Warehouse Cloud data ingestion approaches.

End-To-End Greenfield Scenario

As an example, the Greenfield approach is demonstrated in the following use case (See Figure 5). This means when the customer operates SAP Data Warehouse Cloud together with SAP BW bridge to connect SAP on-premises systems. This provides proven SAP BW-based data integration technology for ABAP-based SAP systems, and enables the rich feature set of extractors in SAP Data Warehouse Cloud.

The SAP BW bridge data, and processes are administered, and managed via an SAP UI5 environment called SAP BW bridge Cockpit. The implementation of new objects is done within Eclipse via SAP BW Modeling Tools, and ABAP Development Tools. Within the SAP Data Warehouse Cloud, the SAP BW bridge artefacts are available via remote tables, and can be used via the SAP Data Warehouse Cloud cross-space sharing approach. An SAP GUI is not required to access the SAP BW bridge environment.

SAP BW bridge: Development-Environment

In the following data flow (see figure 6) it can be seen, that the Eclipse environment with SAP BW Modeling Tools, and ABAP Development Tools are used. The well-known SAP flight data model is the foundation for this use case.

In this scenario, the tables “Flight” for transaction data, and “Airline carrier” for master data are considered. The left branch of the data model handles transaction data, the data is loaded from an SAP ECC system respectively an SAPI Extractor. The right branch of the data model handles master data, the data is loaded from a CDS view. As it can be seen, there is still the subdivision into master data texts, and master data attributes. Within the data flow, transformations, and data transfer processes are used to load data into the advanced DataStore Object, and the master data-bearing InfoObject. Within the Composite Provider, the data is then combined with a join.

The SAP BW bridge component is primarily intended for ODP-based source systems. In this regard, customers have the option to create source systems in the context of ODP. This means that ODP-BW, ODP-HANA, ODP-SAP, ODP-CDS, and ODP-SLT based source systems can be connected. This offers the additional benefits of the Operational Data Provisioning Framework, such as Extract Once Deploy Many, data compression, and more.

Dedicated process chains are created for both branches in order to load data into the InfoProviders. The process chains are modelled in the SAP UI5 environment for SAP BW bridge called SAP BW bridge Cockpit (see figure 7).

SAP BW bridge Space in SAP Data Warehouse Cloud

If a customer wants to use SAP BW bridge, a provisioning process is triggered by SAP. With that, there will be a dedicated Space for SAP BW bridge in the SAP Data Warehouse Cloud Tenant itself, generated by the provisioning process. This space then has a specific type “SAP BW bridge” (see figure 8).

In the generated SAP BW bridge Space, a connection (see figure 9a) to the SAP BW bridge environment, within the SAP Business Technology Platform, will be generated that contains an SAP HANA Cloud Smart Data Access endpoint, and a HTTP ABAP endpoint. It is possible to connect only one SAP BW bridge System to an SAP Data Warehouse Cloud Tenant.

The SAP HANA Cloud Smart Data Access endpoint is used to connect to the external schema of SAP BW bridge’s SAP HANA Cloud Database that contains the read-only views to the data tables, for moving over the data. The HTTP ABAP endpoint is used to be able to call Monitor UIs via Single Sign on with a named user, and to get Meta Data e.g., for a value help or import of SAP BW bridge Objects (see figure 9b).

The new connection type cannot be edited by a user in the SAP BW bridge Space, as this connection will be generated by the SAP BW bridge provisioning process automatically. The credentials for the SAP HANA Cloud Smart Data Access connection are provided when the connection is generated. The data tables of the Business Technology Platform environment for SAP BW bridge are exposed as remote tables in the SAP BW bridge Space.

Important: The SAP BW Service Key should be copied, as this needs to be entered when an SAP BW bridge project is set up in Eclipse.

Inside the SAP BW bridge Space, the Create-Button for new connections is disabled, as this Space is restricted to the SAP BW bridge only. The Real-Time Replication Status is inactive for this connection, as it only allows Remote Tables (see figure 9c).

SAP BW bridge Space: Data Builder for importing remote tables

The main purpose of the Data Builder regarding the SAP BW bridge Space (see figure 10) is to import, and share the remote tables for other Spaces, using the Cross-Space Sharing Approach of SAP Data Warehouse Cloud. Unlike the regular Spaces of SAP Data Warehouse Cloud, it is not possible to create Tables, Graphical Views, SQL Views, Entity Relationship Models, or Data Flows within the Data Builder of a SAP BW bridge Space. SAP Data Warehouse Cloud artefacts using the SAP BW bridge remote tables can only be created in other spaces based on the shared remote tables.

Using the Import button in the Data Builder, then via “Import Remote Tables”, the tables of SAP BW bridge InfoProvider can be accessed via the underlying connection (see figure 11).

In the “Import Remote Tables” wizard there is only one connection available, the connection to the SAP BW bridge System (see figure 12). By selecting the defined connection, the connection gets validated. If the validation process is successful, the next step is available.

The wizard for the data tables of SAP BW bridge InfoProvider contain the following data tables, which are then available as Remote Tables in SAP BW bridge Space in SAP Data Warehouse Cloud itself.

• Advanced DataStore Object (Reporting View)
• Composite Provider
• Master Data Tables
  • Attributes
  • Texts

The data tables are displayed by InfoAreas (see figure 13). It is possible to multi select tables to support a mass take over. It is also possible to select an entire InfoArea, then all the tables underneath the objects are selected. Afterwards it is possible to deselect some tables (then the InfoArea can be deselected).

The last step displays the list of objects which are ready for import. There is one more section for Remote Tables, which are already in the repository of SAP Data Warehouse Cloud. The user can also change the technical name, and the business name of the appropriate object. Via “Import and Deploy” the remote tables will be generated with Semantic Usage as Relational Dataset. The Master Data Text tables can be generated as either Dimension or Text (see figure 14).

Next, the remote tables located in the SAP BW bridge Space in SAP Data Warehouse Cloud need to be shared with the regular SAP Data Warehouse Spaces. As already outlined before, the main functionality here is to import, and share the remote tables for other spaces. SAP Data Warehouse Cloud artefacts using the remote tables can only be created in other spaces, based on the shared remote tables.

In my example, I have created a standard SAP Data Warehouse Cloud Space “DENIZBWBRIDGE”, which consumes the artefacts, and allows further implementation within the SAP Data Warehouse Cloud (see figure 15).

SAP Data Warehouse Cloud Space: Consuming Shared SAP BW bridge Artefacts

Within standard SAP Data Warehouse Cloud Spaces, the shared SAP BW bridge Remote Tables can be accessed, and other SAP Data Warehouse Cloud functionality can be applied accordingly (see figure 16).

The following SQL code of the previous graphical view (see figure 17) states that the view within the standard SAP Data Warehouse Cloud Space accesses the remote tables of the SAP BW bridge Space in SAP Data Warehouse Cloud.

SELECT *
FROM (("BWBRIDGEDEMO.ZDOFLIGHTREPORTING" AS "ZDOFLIGHTREPORTING"
INNER JOIN "BWBRIDGEDEMO.ZDO_AIRLATTRIBUTES" AS "ZDO_AIRLATTRIBUTES" 
ON "ZDOFLIGHTREPORTING"."CARRID" = "ZDO_AIRLATTRIBUTES"."ZDO_AIRL") 
INNER JOIN "BWBRIDGEDEMO.ZDO_AIRLTEXT" AS "ZDO_AIRLTEXT" 
ON "ZDO_AIRLATTRIBUTES"."ZDO_AIRL" = "ZDO_AIRLTEXT"."ZDO_AIRL");

Figure 17: SQL-Code of Graphical View

SAP Analytics Cloud: Story based on SAP BW bridge data

Finally, based on the analytical data set of SAP Data Warehouse Cloud, which in this case processes SAP BW bridge data, a visualisation of the data can be done via SAP Analytics Cloud (see figure 18) or any other 3rd party front end solution.

Like for other data models in SAP Data Warehouse Cloud you can use SAP Data Warehouse Cloud Live Data Connection of SAP Analytics Cloud. However, SAP Analytics Cloud generally has certain limitations with SAP Data Warehouse Cloud Live Data Connection, detailed information is available in SAP Note 2832606.

SAP BW bridge Space: Data Integration Monitor

Within the SAP BW bridge Space only Remote Tables are available. For the Data Integration Monitor of the SAP BW bridge Space that means, that the View Persistency Monitor, and Data Flow Monitor are not visible (see figure 19).The available functionalities here are the Remote Table Monitor, and Remote Query Monitor. In addition, access to the SAP BW bridge Cockpit is possible via the Data Integration Monitor of the SAP BW bridge Space.

The post SAP Data Warehouse Cloud, SAP BW bridge: Overview and Technical Deep Dive appeared first on ERP Q&A.

Method 1 – Traditional Type 2 SCD

This method is the traditional Type 2 SCD approach that creates a new entry in the dimension table whenever a change has been identified. Valid from and to dates will be updated on the old and new record to reflect the record history.

Pro’s

• Most development effort required

This method will require the most amount of development effort and complexity as it is necessary to compare the source data with the existing data in the dimension table to identify the record status as ether new, existing or deleted. Once this has been determined, existing records must be evaluated to establish whether a change has occurred. The entire dataset must then be recreated with the appropriate valid to/from dates and flags.

• Most storage-efficient method

As this method only captures when there is a change, it requires the least of amount storage.

• Greater reporting flexibility

With the use of valid from/to dates and deleted dates this method provides you with the most reporting flexibility

Con’s

• 2 Dataflows are required -Time staggered

As historical entries require updating, and currently there is only the method to append\truncate when using a dataflow, the entire dataset will need to be recreated using truncate. Due to the table being truncated, it is necessary to use a temporary table to load the data into first, before truncating the main dimension table. This means that the use of 2 dataflows is required. Currently it is not possible to link Dataflow tasks to run in sequence so they will need to be scheduled with a sufficient time delay between each job.

Method 2 – Daily Snapshot (Append)

This method captures a snapshot of the dimension on a daily basis by adding a Snapshot Date to the Dimension. The data is appended to the Dimension table; so if the load was run multiple times on a day, only the first run would be captured.

Pro’s

• Minimal development effort required

This method only requires one extra column to be populated with the snapshot/load date.

• Only 1 Dataflow required

As only one table is required for this method there is only a need for one dataflow

Con’s

• Data will be duplicated / Dataset size needs to be considered

Creating Snapshots will mean duplication of data within the Dimension table so we also need to consider the size of the dataset as the table could become extremely large if being appended every day.

• Snapshot date usage

You need to consider how to use the Snapshot date within your data model to ensure you don’t get duplicated records when joining to the dimension.

Method 3 – Daily Snapshot (Truncate)

This method is the same as Solution 2 as it captures a daily snapshot of the dimension, the main difference being that the entire dimension is recreated resulting in the most recent values being retained.

Pro’s

• 2 Dataflows are required – Time Staggered

As the dataset is truncated but also used as a source, this method requires the use of two dataflows that will need to be scheduled with a time delay between them.

Conclusion

Hopefully these three methods will help you if you have a similar modelling requirement in Data Warehouse Cloud. Each has its own advantages and disadvantages so you will obviously need to decide which is most appropriate for you.

Method 1 (Traditional Type 2 SCD) was the chosen method to implement for the DWC project I was working on due to the large volume of data and low volume data changes.

There are a number of upcoming items in the Data Warehouse Cloud Roadmap which may well help achieve the same objective but it’s not yet clear when they will arrive but please monitor as they might be beneficial.

Expected – Q4 2021

Roadmap item – Dependent serial scheduling tasks

Anticipated Impact – Expected to provide the ability to chain task\dataflows together

URL – https://roadmaps.sap.com/board?PRODUCT=73555000100800002141&range=FIRST-LAST#;INNO=000D3A4855C81EDB99C49210E6AFB1BD

Expected – Q4 2021

Roadmap item – Time-dependency support added to dimensions

Anticipated Impact – Expected to provide the support for a Type 2 SCD with Valid From\To

URL – https://roadmaps.sap.com/board?PRODUCT=73555000100800002141&range=FIRST-LAST#;INNO=000D3ABE772D1EDBB6DEA7546CD6AFB3

If you got this far then thanks for reading my first blog and I hope you have found it useful, please post any questions, comments and feedback that you may have below and don’t forget to follow me!

If you are interested in implementing one of the methods then please take a look at the Solution Walkthroughs below for a more detailed step-by-step guide and I will try to answer any questions or comments.

Solution Walkthroughs

This section details the individual steps needed to implement each of the solutions

1. Solution 1 – Traditional Type 2 SCD

The first solution is a traditional Type 2 Slowly Changing Dimension where any change in a record will create a new entry and the valid from\to dates updated accordingly.

Below is a high-level overview of all the objects used in the solution with a short description of the object usage.

In the Data Builder there are five different object types available and currently no way to distinguish if a View is Graphical\SQL once created. Applying a naming convention such as the one below helps aid clarity.

1.1 Naming Convention

• Tables – Prefixed with “TB_”
• Views (Graphical) – Prefixed with “VG_”
• Views (SQL) – Prefixed with “VS_”
• Dataflow – “DF_”
• Entity Relationship Model – “ER_”

1.2 Step 1 – Create Source

Create a Source Table\View to use as the source for the Dimension (“TB_Source_CSV”).

1.3 Step 2 – Create Dimension tables

Create two tables (“TB_Dim_SCD_Temp” and “TB_Dim_SCD”). These tables we be identical and will match the source plus five additional fields:

• Valid From
• Valid To
• Is Current
• Is Deleted
• Deleted Date

1.4 Step 3 – Create VG_Dim_SCD_1 – Identify Change Type

Create a new Graphical View to identify data changes

Add the Source Table\View twice (“TB_Source_CSV”) and alias them as:

• Source Data
• Source Current

Now add the table “TB_Dim_SCD” three Times and alias them as:

• Dim Current
• Dim Not Current
• Dim Deleted

Add a Filter to Dim Current that checks:

• Is_Current = 1 and Is_Deleted =

Add a Filter to Dim Deleted that checks:

• Is_Current = 1 and Is_Deleted = 0

Add a Filter to Dim Not Current that checks:

• Is_Current = 0 or Is_Deleted = 1

Left Join Source Data and Dim Current ensuring the Source Data is the left\driving table and the tables are joined on the primary key.

A Projection should be automatically added after the join, if it is not, add a projection.

Enable all fields in the Projection.

Rename all duplicated fields from Dim_Current

Add a calculated column.

Add a new field called SCD (String (10)).

Create a Case Statement:

If key Dim field is null Then New

If any Non-key fields has changed value Then Updated

Else No Change

Go back to Dim Not Current.

Add a Calculated column after the filter.

Add a new field called SCD (String (10)).

Hardcode the value “No Change”.

Left Join Source Deleted and Dim Deleted ensuring the Dim Deleted is the left\driving table and the tables ae joined on the primary key.

A Projection should be automatically added after the join, if it is not, add a projection.

Disable all fields from Source Deleted except the Key field.

Rename the Key field from Source Deleted.

Add a filter that returns where the Source Deleted Key field is null.

Add a calculated column.

Add a new field called SCD (String (10)).

Hardcode the value No Change.

The three datasets are now ready to be combined with a Union operator.

Start with Current and Not Current then Add Deleted.

Ensure all fields have been correctly mapped in the union.

The first view is now completed and it can be identified if the Dimension data is new, changed, deleted or unchanged.

1.5 Step 4 – Create VG_Dim_SCD_2 – Recreate Dimension

Create a New Graphical View

Add “VG_Dim_SCD_1” 5 times to the view and alias as followed:

• New
• Updated Historic
• Updated New
• Deleted
• No Change

Add a filter to each flow using SCD based on the alias name:

• New = New
• Updated Historic = Updated
• Updated New = Updated
• Deleted = Deleted
• No Change = No Change

Add a Calculated Column transform to each flow and derive the values to be used for:

• Valid From
• Valid To
• Is Current
• Is Deleted
• Deleted Date

Use the values below as the derived values for each alias

New

• Valid From = ‘1900-01-01’ – As your history only starts from the point that the load starts we use 1900 instead of the run date to cater for historic reporting
• Valid To = ‘9999-12-31’
• Is Current = 1
• Is Deleted = 0
• Deleted Date = NULL

No Change

• Valid From = Dim Valid From
• Valid To = Dim Valid To
• Is Current = Dim Is Current
• Is Deleted = Dim Is Deleted
• Deleted Date = Dim Deleted Date

Deleted

• Valid From = Dim Valid From
• Valid To = Dim Valid To
• Is Current = Dim Is Current
• Is Deleted = 1
• Deleted Date = Current\Load Date

Updated New

• Valid From = Current\Load Date
• Valid To = ‘9999-12-31’
• Is Current = 1
• Is Deleted = 0
• Deleted Date = NULL

Updated Historic

• Valid From = Dim Valid From
• Valid To = Current\Load Date
• Is Current = 0
• Is Deleted = 0
• Deleted Date = NULL

Add a Projection to each flow.

At this point it is important to ensure the correct fields are being used for all attributes.

Exclude all columns.

Enable Valid From, Valid To, Is Current, Is Deleted and Deleted Date

New – Enable all Fields from Source Data

No Change – Enable all Fields from Source Data\Dimension Data

Deleted – Enable all Fields from Dimension Data

Updated New – Enable all Fields from Source Data

Updated Historic – Enable all Fields from Dimension Data

Union all datasets

1.6 Step 5 – Create DF_Dim_Temp_Load – Load into Temp Table

The reason for using Dataflows\Tables is because the target table is also used as the source and the target is truncated when loaded.

When a dataflow is executed, the first step is to truncate the target table, this results in all records falling into the SCD category new and all history being lost.

To avoid this issue, first load the data into a temporary table.

Use “VG_Dim_SCD_2” as a Source and “TB_Dim_SCD_Temp” as the target set the table-loading mode to TRUNCATE

1.7 Step 6 – Create DF_Dim_Load – Load from Temp Table to Dimension

Use ”TB_Dim_SCD_Temp” as a Source and “TB_Dim_SCD” as the target set the table-loading mode to TRUNCATE

1.8 Step 7 – Schedule Dataflows

The Data Integration Monitor doesn’t currently have the ability to link/chain Dataflow executions, you only have the ability to schedule based on time\frequency.

Ensure that you give ””DF_Dim_Temp_Load” enough time to complete the execution prior to the “DF_Dim_Load” schedule starting.

2 Solution 2 – Daily Snapshot (Append)

Instead of comparing the data, this solution takes a daily snapshot of the entire dataset and appends this to the Dimension table with a Snapshot Date.

2.1 Naming Convention

• Tables – Prefixed with “TB_”
• Views (Graphical) – Prefixed with “VG_”
• Views (SQL) – Prefixed with “VS_”
• Dataflow – “DF_”
• Entity Relationship Model – “ER_”

2.2 Step 1 – Create Source

Create a Source Table\View that you would like to use as your source for the Dimension “TB_Source_CSV”

2.3 Step 2 – Create Dimension table

Create the table “TB_Dim_SCD” this will be identical to the source plus one additional field

• Snapshot Date

2.4 Step 3 – Create VG_Dim_SCD_1 – Check Dimension Max Snapshot Date

Create a new Graphical View

Add “TB_Source_CSV” to the design pane add alias as Source

Add “TB_Dim_SCD” to the design pane add alias as Dim

The aggregation function only aggregates on Integer data types, so we will create an integer representation of the date

Add a calculated column transform to both flows and add the following fields

• Source
  
  • Snapshot Date – Integer – CURRENT_DATE())
    
  • Snapshot Date INT – Integer – DAYS_BETWEEN(‘1900-01-01’,CURRENT_DATE())
    
• Dim
  
  • Snapshot Date INT – Integer – DAYS_BETWEEN(‘1900-01-01’, “Snapshot Date”)

Add a Projection to the Dim flow

Disable all fields excluding Snapshot Date INT

Add an aggregation transform

Set Snapshot Date INT to MAX aggregation type

Join the two flows together

Set the Join Type to Cross Join and remove any mappings

In the projection rename Snapshot Date INT (Dim) to Max Snapshot Date and ensure all fields are enabled

Add a calculated column transform

Update the formula in Max Snapshot date to replace null with 0 to cater for loading into an empty Dimension table

Add a filter where Snapshot Date INT > Max Snapshot Date INT

2.5 Step 4 – Create\Schedule Dataflo

Create a dataflow to load the data from your view to the Dimension, ensure the table load type is set to APPEND and schedule the Dataflow.

3 Solution 3 – Daily Snapshot (Truncate)

This solution is a replica of Solution 2, the difference here is that rather than checking the Max Snapshot Date in the Dimension, all the data is extracted from the Dimension except where the Snapshot Date equals the Load Date.

This allows the load to run multiple times in a period and the most recent run in the period will be the version held in the Dimension.

As the target table is truncated, two Dataflows and a temporary table will be required in this version of the solution.

3.1 Naming Convention

3.2 Step 1 – Create Source

Create a Source Table\View that you would like to use as your source for the Dimension “TB_Source_CSV”

3.3 Step 2 – Create Dimension tables

Create the tables “TB_Dim_SCD” and “TB_Dim_SCD_Temp”, they will be identical to the source plus one additional field

• Snapshot Date

3.4 Step 3 – Create VG_Dim_SCD_1 – Combine Historic and Current Dimension

Create a new Graphical View

Add “TB_Source_CSV” to the design pane add alias as Source

Add “TB_Dim_SCD” to the design pane add alias as Dim

Add a calculated column transform to the source flow and add the following fields

• Source
  • Snapshot Date – Integer – CURRENT_DATE())

Add a Filter to the Dim flow where Snapshot Date < CURRENT_DATE())

UNION the two flows together

Set the Join Type to Cross Join and remove any mappings

3.5 Step 4 – Create DF_Dim_Temp_Load

Create a dataflow to load the data from your view to the temporary table, ensure the table load type is set to TRUNCATE.

3.6 Step 5 – Create DF_Dim_Load

Create a dataflow to load the data from your temporary table to the Dimension table, ensure the table load type is set to TRUNCATE.

3.7 Step 6 – Schedule Dataflows

Ensure that you give ”DF_Dim_Temp_Load” enough time to complete the execution prior to the “DF_Dim_Load” schedule starting.

The post SAP Data Warehouse Cloud – How to Create a Slowly Changing Dimension appeared first on ERP Q&A.