Real-time data ingestion (CDC) from SAP to Snowflake
The world is full of paradoxes. Here’s one for data analytics professionals: analytics becomes pervasive when it disappears.
For decades, business intelligence (BI) and analytics tools have failed to penetrate more than 25% of an organization. And within that 25%, most workers use the tools only once or twice a week. Embedded analytics changes the equation. By inserting charts, dashboards, and entire authoring and administrative environments inside other applications, embedded analytics dramatically increases BI adoption. The catch is that most business users don’t know they’re “using BI”—it’s just part of the application they already use. The best BI tools are invisible.
By placing analytics at the point of need—inside operational or custom applications—embedded analytics closes the last mile of BI. Workers can see the impact of past actions and know how to respond to current issues without switching applications or context. Analytics becomes an indispensable part of the way they manage core processes and solve problems. As a result, embedded analytics has a much higher rate of adoption than traditional BI or analytics.
Embedded analytics has much higher rate of adoption than traditional BI or analytics.
Embedded analytics makes existing applications more valuable for every organization. Independent software vendors (ISVs) say that embedded analytics increases the value of their applications and enables them to charge more for them. Enterprise organizations embed analytics into operational applications, such as Salesforce.com, and intranet portals, such as SharePoint. In both cases, embedded
analytics puts data and insights at users’ fingertips when they need it most—both to gain insights and take action.
ISV requirements. In the embedded world, ISVs have more stringent requirements than traditional organizations. (See “Twelve Evaluation Criteria” below.) ISVs must ensure an embedded product looks and feels like their host application, and thus require greater levels of customization and extensibility.
Also, cloud ISVs require embedded products that work in multi-tenant environments, with seamless user administration and custom deployments. Many ISVs offer tiered pricing, which requires embedded products with flexible user provisioning. Finally, because ISVs can’t always estimate how many customers will purchase the analytics, they need flexible and affordable pricing models.
Enterprise requirements. Traditional enterprises have fewer requirements than ISVs, but that is changing. More companies are pursuing digital strategies that require customer-facing Web applications. And although most don’t charge for analytics, as ISVs do, many view data analytics as a key part of the online customer experience. For example, mutual funds now provide customers with interactive dashboards where they can slice and dice their portfolios and take actions such as buying and selling funds. Thus, their requirements for customization, extensibility, multi-tenancy, and security have grown significantly in recent years.
Once organizations decide to embed analytics, they need to make a few key decisions. The first is whether to build their own analytics or buy a commercial off-the-shelf tool.
Build. Organizations with internal developers are always tempted to build their own analytics. But
unless the analytics are simple and users won’t request changes, it’s always smart to outsource analytics to a commercial vendor. Commercial analytics products deliver best-of-breed functionality that would take in-house developers years to develop, distracting them from building the host application.
Buy. Many analytics vendors have made their tools more configurable, customizable, and integrateable with host applications. Most see embedded analytics as a big market and aim to make their tools blend seamlessly with third-party applications. They also make it easy to customize the tool without coding, including changing the graphical user interface (GUI) or the ways users navigate through the tool or interact with its components. When extensive customization or integration is required, customers can use application programming interfaces (APIs) to fine-tune the tool’s look and feel, create new data connectors, charts, event actions, and export types.
The second decision is to figure out the architecture for embedding analytics. From our research, we’ve
discovered three primary approaches. (See figure 1.)
1. Detached analytics. This is a lightweight form of embedding where the two applications—host and analytics—run separately but are tightly linked via URLs. This approach works well when multiple applications use the same analytics environment, portal, or service. A common example is Google Analytics, a commercial service that multiple groups inside an organization might use to track Web traffic on various internal websites. There is no commonality between the host application and analytics tool, except for a shared URL and shared data. The two applications might also share a common authentication mechanism to facilitate single signon (SSO). Ths approach is rather uncommon these days.
2. Inline analytics. With inline analytics, output from an analytics tool is embedded into a host application—it looks, feels, and acts like the host but runs as a separate element, tab, or module within it. For example, a newspaper might embed a chart within the text of an article on a webpage. Or an ERP application might present users with a dashboard upon logging in that displays summary activity from each module in the application. Or there might be a separate tab where customers can view analytics about their activity within the application. In most
cases, the embedded components sit within an iFrame, which is an HTML container that runs inside a webpage. iFrames were once the predominant method for embedding analytics content but are disappearing due to security and other concerns. (See next section.)
3. Fused analytics. Fused analytics delivers the tightest level of integration with a host application. Here, the analytics (e.g., a chart, table, or graphical component) sit side by side with the host application components and communicate bi-directionally with them. This created a “fused” or integrated environment where the end users aren’t aware that a third-party tool is part of the experience.
For example, an inventory manager can view inventory levels in various warehouses and place replenishment orders without leaving the screen. Or a retail store manager can view daily demand forecasts and then click a button to create the shift schedule for the following week. Fused analytics is facilitated by JavaScript libraries that control front-end displays and REST API calls that activate server functions. Analytics tools with extensive API libraries and programming frameworks make all their functionality available within a host application, including collaboration capabilities, alerts, reporting and augmented analytics features.
Most analytics vendors can support inline analytics without much difficulty. They simply provide “embed code”—a snippet of HTML and JavaScript—that administrators can insert into the HTML code of a webpage. The embed code calls the analytics application to display specified content within an iFrame on the webpage. People who use YouTube and other social media services are familiar with embed code.
iFrames are a quick and easy way to embed third-party content, and most analytics vendors support them.
But iFrames have disadvantages. Because they are frames or windows inside a webpage that are controlled by an external application or service, they pose considerable security risks. Also, they operate independently of the host webpage or application—the host can’t manipulate what’s inside the iFrame, and vice versa. For example, hitting the back button doesn’t change what’s inside the iFrame.
Furthermore, iFrames behave differently depending on the browser, making them difficult to manage. Consequently, a growing number of organizations refuse to allow iFrames, and the software industry is moving away from them Fused analytics requires tight integration between an analytics and host application.
Fused analytics also requires a much greater degree of customization, extensibility, flexibility, and integration than many analytics vendors support out of the box. To simplify fused analytics, many BI vendors have wrapped their APIs in programming frameworks and command line interfaces (CLIs) that make it easy for programmers to activate all functions in the analytics tool. These Javascript frameworks and CLIs have been a boon to analytics embedding. Nonetheless, companies that want to fuse analytics into an application need to look under the covers of an analytics tool to discover its true embeddability. (See “Select a Product” below.)
Another major decision is selecting a commercial analytics product to embed. Selecting a product that doesn’t include a key feature you need, such as alerts or exports to PDF or printing, can undermine adoption and imperil your project. Or maybe the product doesn’t work seamlessly in a multi-tenant environment, making administration cumbersome and time-consuming and contributing to errors that undermine customer satisfaction. Or your deployment drags out by weeks or months because most customizations require custom coding.
This report is designed to help you avoid these and other pitfalls of embedded analytics. Whether you are an independent software vendor (ISV) that needs to know how to embed analytics in a commercial, multi-tenant cloud application or the vice president of application development at major corporation who wants to enhance a homegrown application, this report will provide guidance to help you ensure a successful project.
The report outlines a four-part methodology:
The report’s appendix drills into the evaluation criteria in depth, providing questions that you should ask
prospective vendors to ensure their products will meet your needs.
Many teams start their embedded analytics project by selecting an analytics product to deploy. Although choosing the vendor to power your analytics is a critical step, it shouldn’t be the first milestone you tackle. Instead, start by considering these questions: What are you trying to build, for whom, and why? These essential questions will help you better understand your product goals and will aid in selecting the
best tool to achieve your goals.
Start by asking: What are you trying to build, for whom, and why?
We recommend a six-step model to ensure that your analytics are not only technically successful, but achieve your business goals
Setting the goals for your analytics project is an essential first step to ensure that all key stakeholders—from the executive team to the end users—are fully satisfied upon project completion.
There are three basic steps to planning for a successful analytics project: define table stakes, define delighters, and define what’s out of scope.
The composition of the project team can be a key element in the success or failure of an analytics project. For example, more than a few projects have been derailed at the last minute when the legal team—not included in the process—surfaced major issues with the plan. When structuring your analytics product team, consider including the following roles in your “core team” for the project:
> Product owner/head of product
> CTO
> Head of development/engineering
> Head of sales
> Head of marketing
> Head of operations and support
Next, identify roles that, although not involved in daily decisions, will need to be consulted along the
way, including finance, billing, legal, and sales enablement/training.
A best practice is to get the key project stakeholders in a single room to discuss core elements in a facilitated session. Although it might be necessary to have some participants attend remotely, in-person attendance makes for a faster and more effective session.
Too often project teams—whether analytics-focused or otherwise—fail to create a plan to guide the key steps required to bring embedded analytics to market. Without a plan, teams are liable to spend too much time gathering requirements and too little time analyzing persona needs. Without planning, the time required to perform key tasks, such as resolving issues from beta testing, might be overlooked. The
steps to creating a basic, but useful, plan are simple:
Set project goals. Setting project goals before any technical work starts is a good way to ensure that everyone involved agrees on the success criteria. Set aside time to create project goals as the first step in your analytics plan.
Set a timeline. A timeline may seem obvious, but it’s important that, in addition to the overall start and end dates, you plan for intermediate milestones such as:
It’s easy to forget that although you, as a member of the product team, might be fully aware of everything that’s taking place within your analytics project, others might not know about your progress. In the absence of information, you might find that inaccurate data is communicated to customers or other interested groups. You can prevent this by establishing a communication plan, both for internal personnel and for potential customers. Although the plans will be different for those inside your walls versus external parties, all communication plans should include:
> Regular updates on progress
> Revisions of dates for key milestones
> Upcoming events such as sneak peeks or training sessions
Once you’ve started your product development effort, particularly once you’ve started beta testing or rollout, it can be hard to identify when critical problems surface. That’s why setting metrics and tripwires is a good idea during the planning phase.
It can be hard to identify when critical problems surface.
That’s why setting metrics and tripwires is a good idea.
Metrics are used to measure product performance and adoption. An embedded product should have
a dashboard that enables ISVs and OEMs to monitor metrics and activity across all tenants using the
product, alerting administrators when performance goes awry. Consider tracking:
> Product uptime
> Responsiveness of charts and dashboards (i.e., load time)
> Data refresh performance and failures
> Number of reloads
> Total users
> Users by persona type
> Number of sessions
> Time spent using the analytics
> Churn (users who don’t return to the analytics application)
> Functionality used, e.g., number of reports published, alerts created, or content shared
Tripwires alert you to critical situations before they cause business-disrupting problems. They are metrics values that, if exceeded, trigger a response from the development team. As an example, you might have a tripwire that states if the product uptime is less than 99.9%, the general rollout of the analytics product will cease until the issue is resolved. Each metric should have an established tripwire, and each tripwire should contain the following elements:
> A metric value that, if exceeded, triggers a response.
> A predetermined response such as “stop rollout” or “roll back to the previous version.”
> A responsible party who reviews the metric/tripwire and determines if action is required.
Although metrics and tripwires don’t ensure project success, they can greatly reduce the time —and stress for the team—to address problems.
It’s a common mistake to think either that you fully understand the users’ needs or that all users are the same. Many teams launch embedded analytics products without considering the detailed needs of target users or even the different user types they might encounter. Avoid this situation by creating detailed user personas and doing mission/workflow/gap analysis.
Many teams launch embedded analytics products without considering the detailed needs of their users or even the different user types they might encounter.
Here’s how it works:
Step One: Choose personas. The best way to create an engaging data product is to deliver analytics that solve users’ problems. This is difficult to do for a generic “user,” but it can be accomplished for a specific user “persona” or user type. Start by picking two or three key user types (personas) for whom you will tailor the analytics. These might be strategic users looking for patterns and trends (like executives) or tactical users focused on executing work steps (like salespeople or order fulfillment teams). Although you may ultimately add functionality for many personas to your analytics, start with personas that can impact adoption—key decision makers—first. Get these user types engaged with your analytics application and they can help drive adoption among other users.
Step 2: Identify mission. For each chosen persona, the next task is to understand the user’s mission. What is the person trying to accomplish in their role? Are they trying to improve overall sales? Are they striving to increase revenue per customer? Understanding what the persona must accomplish will help you understand where analytics are needed and appropriate cadence.
Step 3: Map workflows and gaps. Now that you understand each persona’s mission, the third step is to outline the workflow they follow and any gaps that exist. These steps—and gaps—inform the project team where they can add analytics to assist the persona in accomplishing their mission. Keep it simple. If your persona is “head of sales” and the mission is “increase sales,” a simple workflow might be: (a) review sales for the month (b) identify actions taken within those segments (c) recommend more effective tactics to managers.
Within this workflow, you might find opportunities where analytics can improve the effectiveness of the head of sales. Perhaps reviewing sales performance or identifying underperforming segments requires running reports rather than simply reviewing a dashboard. Maybe seeing what actions have been taken requires investigating deep within the customer relationship management (CRM) system and correlating actions back to segments.
By finding information gaps within workflows and understanding personas’ missions,
project teams can ensure they put high-value analytics in front of users.
By finding information gaps within workflows and understanding personas’ missions, project teams can ensure that they put high-value analytics in front of users. It becomes less of a guessing game—replicating existing Microsoft Excel-based reports and hoping the new format attracts users—and more of a targeted exercise. Only analytics that truly add value for the persona are placed on the dashboard, in a thoughtful layout that aids in executing the mission. Engagement increases as target users solve problems using analytics.
Once you’ve defined user requirements, you need to turn them into specifications for selecting a product. The following evaluation criteria will help you create a short list of three or four vendors from the dozens in the market. The criteria will then guide your analysis of each finalist and shape your proof of concept.
We’ve talked with dozens of vendors, each with strengths and weaknesses. Analyst firms such as Gartner and Forrester conduct annual evaluations of Analytics and BI tools, some of which are publicly available on vendor websites. G2 provides crowdsourced research on BI tools, while the German research firm BARC publishes a hybrid report that combines analyst opinions and crowdsourced evaluations.
However, these reports generally don’t evaluate features germane to embedded analytics. That’s because the differentiators are subtle and often hard to evaluate, since it requires diving into the code.
The differentiators among embedded analytics are subtle and often hard to evaluate since it requires diving into the code.
For companies that want to tightly integrate analytics with a host application, there are three key things to look for:
> How quickly can you deploy a highly customized solution?
> How scalable is the solution?
> Does the vendor have a developer’s mindset?
Deployment speed. It’s easy to deploy an embedded solution that requires minimal customization. Simply replace the vendor logo with yours, change font styles and colors, copy the embed code into your webpage, and you’re done. But if you want a solution that has a truly custom look and feel (i.e., white labeling), with custom actions (e.g., WebHooks and updates), unique data sources and export formats,
and that works seamlessly in a multi-tenant environment, then you need an analytics tool designed from the ground up for embedding.
The best tools not only provide rich customization and extensibility,
but they also do so with minimal coding.
The best tools not only provide rich customization and extensibility, but they also do so with minimal coding. They’ve written their own application so every element can be custom-tailored using a pointand-click properties editor. They also provide templates, themes, and wizards to simplify development and customization. And when customers want to go beyond what can be configured out of the box, the
tools can be easily extended via easy-to-use programming frameworks that leverage rich sets of product APIs that activate every function available in the analytics tool.
Moreover, the best embeddable products give host applications unlimited ability to tailor analytic functionality to individual customers. This requires analytics tools to use a multi-tenant approach that creates an isolated and unique analytics instance for each customer. This enables a host company to offer tiered versions of analytic functionality to customers, and even allow customers to customize their analytics instance. This mass customization should work whether the host application uses multitenancy and/or containerization or creates separate server or database environments for each customer.
Scalability. It’s important to understand the scalability of an analytics solution, especially in a commercial software setting where usage could skyrocket. The tool needs strong systems administration capabilities, such as the ability to run on clusters and support load balancing. It also needs a scalable database—whether its own or a third party’s—that delivers consistent query performance as the number of concurrent users climbs and data volumes grow. Many vendors now offer in-memory databases or caches to keep frequently queried data in memory to accelerate performance. The software also must be designed efficiently with a modern architecture that supports microservices and a granular API. Ideally, it works in a cloud environment where processing can scale seamlessly on demand.
Developer mindset. When developers need to get involved, it’s imperative that an analytics tool is geared to their needs. How well is the API documented? Can developers use their own integrated development environment, or must they learn a new development tool? Can the tool run on the host application server or does it require a proprietary application server and database? How modern is the tool’s software architecture? Does it offer JavaScript frameworks, which help simplify potentially complex or repetitive tasks by abstracting API calls and removing the need for deep knowledge of the analytics tool’s APIs by your developers?
Companies are adopting modern software architectures and don’t want to
pollute them with monolithic, proprietary software from third parties
Increasingly, companies are adopting modern software architectures and don’t want to pollute them with monolithic, proprietary software from third parties. The embedded analytics solutions of the future will insert seamlessly into host code running on host application and Web servers, not proprietary servers and software.
It’s important to know what questions to ask vendors to identify their key differentiators and weak spots. Below is a list of 12 criteria to evaluate when selecting an embedded analytics product. (See the appendix for a more detailed description of each item.)
These criteria apply to both ISVs and enterprises, although some are more pertinent to one or the other. For instance, customization, extensibility, multi-tenancy, and pricing/packaging are very important for ISVs; less so for enterprises.
With a plan and tool selected, the next step is to begin development. But perhaps not the development that might initially come to mind. We recommend that, alongside the technical implementation of your analytics, you develop the business aspects of your project. This phase requires you to fully consider how the analytics will be introduced to the market—how they will be packaged, priced, and supported post-launch.
Start by designing the packaging for your analytics. Packaging is particularly important for software vendors who sell a commercial product. But enterprises that embed analytics into internal applications can also benefit from understanding these key principles.
Teams often consider analytics to be an “all-or-nothing” undertaking. You either have a complete set of analytics with a large set of features, or you don’t have any analytics at all.
But this approach fails to consider different user needs. Expert users may desire more sophisticated analytics functionality, while novice users may need less. The “all or nothing” approach also leaves you with little opportunity to create an upsell path as you add new features. It’s better to segment your analytics, giving more powerful analytics to expert users while allowing other users to purchase additional capabilities as they need them.
You never want to give users every conceivable analytical capability from the outset. Instead, use a tiered model. If you’ve ever signed up for an online service and been asked to choose from the “basic,” “plus,” or “premium” version of the product, you’ve seen a tiered model. Keep it simple, don’t add too many levels from which buyers are expected to choose. For example, you might use the following tiers:
> Basic. This is the “entry level” tier and should be the default for all users. You put these introductory, but still useful analytics in the hands of every single user so that they understand the value of analytical insights. Most organizations bundle these analytics into the core application without charging extra, but they usually raise the price of the core application by a nominal amount to cover costs.
> Plus. These are more advanced analytics, such as benchmarking against internal teams (e.g. the western region vs. the eastern region), additional layers of data (e.g. weather data, economic indicators, or financial data), or the ability to drill deeper into charts. This tier should be priced separately, as an additional fee on top of the core application.
> Premium. The top tier will be purchased for use by power users, analysts, or other more advanced users. Here, you might add in features such as external benchmarking (e.g. compare performance to the industry as a whole), and the ability for users to create entirely new metrics, charts, and dashboards. This will be the most expensive tier.
Architecting your offering in this format has several key benefits for data product owners:
> It doesn’t overwhelm the novice user. Although offering too little functionality isn’t ideal, offering too much can be worse. Upon seeing a myriad of complex and perhaps overwhelming features, users may decide the application is too complicated to use. These users leave and rarely return.
> It provides an upgrade path. Over time, you can expect customers to become more sophisticated in their analysis needs. Bar charts that satisfied users on launch day might not be sufficient a year down the road. The tiered model allows customers to purchase additional capabilities as their needs expand—you have a natural path for user growth.
> It makes it easier to engage users. How can you entice customers to buy and use your data product unless they can see the value that it delivers? Including a “basic” analytics tier with minimal, but still valuable, functionality is the answer. The basic tier can be offered free to all customers as a taste of what they can experience should they upgrade to your advanced analytics tiers.
Unfortunately, not all customers will be satisfied by your analytics product, even if it’s structured into a tiered model. Some will require custom metrics, dashboard structures, and more data. Here are some “add-on” offerings that you can charge extra for:
> Additional data feeds. Although your core analytics product might include common data sources, some customers will require different or more data feeds for their use cases. These might include alternative CRM systems, alternative financial systems, weather, economic data, or even connections to proprietary data sources.
> Customized models. A “custom data model” allows buyers to alter the data model on which the product is based. If a buyer calculates “customer churn” using a formula that is unique to them, support this model change as an add-on.
> Visualizations. Customers often request novel ways of presenting information, such as new charts, unique mobile displays, or custom integrations.
> More data. The product team can augment an analytics application by providing more data: Seven years instead of five, detailed transactional records instead of aggregated data.
Data applications can be complex, and they are often deeply integrated into many data sources. For this reason, you might consider offering services to augment your core analytics product:
> Installation/setup. Offer assistance to set up the analytics product, including mapping and connecting to the customer’s data sources, training in-house support personnel, and assisting with loading users.
> Customization. Offer to create custom charts, metrics, or data models.
> Managed analytics. Occasionally, a data product customer requests assistance in interpreting the analytics results. This can take the form of periodic reviews of the analytics (e.g., quarterly check-ups to assess performance) or an “expert-on-demand” service where the customer calls when they have analysis questions.
The situations above are very different from normal product technical support. Managed analysis services can be a highly lucrative revenue source, but they can also consume more resources than anticipated and skew your business model from a product orientation to a services model.
Pricing your analytics sounds like a simple proposition—far less complex than the technical product implementation—but that’s rarely the case. In fact, we’ve seen more than a few instances where the pricing portion of the project takes longer than the actual analytics implementation. But determining the fees to charge for your data product doesn’t have to be daunting. Here are our guidelines to help you
avoid pricing pitfalls.
Charge for your analytics. Analytics can help users make decisions faster, more accurately, and improve overall process cycle times. Such business improvements have value, and you should charge for providing it. However, if your analytics doesn’t add value, some product teams decide to offer analytics free of charge. When there is an apparent mismatch between the value and the price of an analytics product, the answer is to revisit the persona/mission/workflow/gap step of the development process.
Start early. Setting pricing late in the process is a mistake because once the product is fully defined and the costs are set, the flexibility you have for creating pricing options is severely limited. Start early and craft price points appropriate for each product tier (basic, plus, premium) rather than trying to rework key project elements just before launch day.
Keep it simple. Complicated pricing models turn off buyers. They cause confusion and slow the buying cycle. Limit the add-ons available and include key functions in the price of the core application.
Make the basic tier inexpensive. Keep the basic tier as inexpensive as possible. You want people to try analytics and get hooked so they purchase a higher tier. Roll the extra price into the cost of the core application and ensure that every user has access to the basic tier.
Match your business model. If your core application is priced based on a fee per user per month, add a small percentage to that fee to account for the additional value of analytics. Do not add a new line item called “Analytics Functionality” that jumps out at the buyer. Make analytics a part of what your product does.
Many teams spend significant energy designing analytics, creating dashboards, and thinking through pricing, but most forget to consider support processes. Embedded analytics as part of a customer-facing data product are inherently different from enterprise or “inside your walls” analytics. Data products require marketing plans, sales training, and other processes that will allow a properly designed analytics
application to flourish post-launch. Here’s how to get started planning your application’s supporting processes.
List the Impacted Processes
The first step to getting your supporting processes ready is enumerate exactly what will be impacted. There are two ways to go about this step:
1. Brainstorm the processes. The product team spends about 45 minutes brainstorming a list of potentially impacted processes. This is no different from any other brainstorming session—just be sure that you are listing processes (e.g. “process to intake/resolve/close support tickets”) and not organizational names that are less actionable (e.g., “the operations group”).
2. Work from other product lists. If you are part of an organization that has fielded products in the past, you might already have checklists for “organizational readiness” lying around. If so, the list of support processes developed by another product team might be a great place to start. You’ll find that you need to customize the list a bit, but the overlap should be significant, saving you time.
Here is a list of processes commonly impacted by a new data product:
> Provisioning or installation process
> New user onboarding process
> Trouble ticket or issue management
> User experience processes
> Product road mapping process, including request intake
> Utilization tracking or monitoring
> Sales training process
> Billing process
> Decommissioning or deactivation process
Define Changes to Processes
The next step is to determine the degree to which each of the listed processes might need to change to
support your new data product.
> Create a simple flow chart for each potentially impacted process.
> Apply scenarios. Pretend an issue occurs with your analytics. Can your existing process address it? Add or modify process steps to accommodate the analytics product.
> Publish and train. Present the new processes to any teams that might be impacted and store the process documentation wherever other process documents are kept.
Create Metrics
With new processes in place, you need to monitor process performance to ensure that everything is working as planned. For each new or modified process, create metrics to track cycle times, throughput, failure rate, cost, and customer satisfaction. Benchmark these processes against existing processes to ensure they are performing in parity.
The process isn’t over when you’ve deployed your analytics and brought users on board. In fact, the most successful analytics teams view embedded analytics as a cycle, not a sprint to a finish line. Post-launch, you need to consider what is working and what isn’t; and you need to add or fine-tune functionality to better meet persona needs. You need to continuously improve to ensure that analytics adoption and usage doesn’t decline over time.
People always find unique ways of using analytics functionality. Learn what your users are doing; their experiments can guide your project development. Here are three ways to gather feedback on your analytical application:
> Use analytics on analytics. Some platforms allow you to monitor which dashboards, charts, and features are being used most frequently. Track usage at the individual and aggregate level. What functionality is being used? How often is the functionality reloaded? Number of sessions? New, recurring, one time use counts? Three-month growth by user, tenant, type of user, etc?
> Monitor edge cases. The users who complain the most, submit requests, and call your help desk are a gold mine of information. They will often talk—at length—about what functionality can be implemented to make the analytics better for everyone. Don’t ignore them.
> Shoulder surfing. Shoulder surfing is an easy way to gather insights. Get permission from users to observe them interacting with the analytics product on their own tasks at their own computers in their own environments. Shoulder surfing can uncover incredible insights that users might fail to mention in a formal focus group.
Although you started with a limited number of personas and workflows during the initial implementation, the key to sustaining your analytics is to expand both the personas served and the use cases addressed. If you started with an executive persona, consider adding tactical personas that need information about specific tasks or projects. Also, add workflows for existing personas. For example, add a budgeting dashboard for the CFO to complement the cash flow might analytics previously deployed.
Unfortunately, in the absence of new information, users will assume that no progress is being made. Even if you can’t add all the personas, workflows, and functionality required immediately, make sure to create a communication plan so users understand what’s coming next and for whom.
Embedding another product in your application is not easy. There’s a lot that can go wrong, and the technology is the easy part. The hard part is corralling the people and establishing the processes required to deliver value to customers.
Here are key success factors to keep at the forefront of your mind during an embedded analytics project:
1. Know your strategy. If you don’t know where you’re going, you’ll end up somewhere you don’t want to be. Define the goal for the project and keep that front and center during design, product selection, and implementation.
2. Know your users. Pushing dashboards to customers for the sake of delivering information will not add value. Dashboards, whether embedded or not, need to serve an immediate need of a specific target group. Identify and address information pain points and you’ll succeed.
3. Identify product requirements. It’s hard to tell the difference between embedded analytics tools. Use the criteria defined in this report to find the best product for your needs. It may not be one you already know!
4. Define a go-to-market strategy. Here’s where the wheels can fall off the bus. Before you get too far, assemble a team that will define and execute a go-to-market strategy. Especially if you are an ISV, get your sales, marketing, pricing, support, training, and legal teams together at the outset. Keep them informed every step of the way. Make sure they provide input on your plan.
Following these key principles will help ensure the success of your embedded analytics project
For more information or enquiries about Qlik products and services, feel free to contact us below.
Connect with SIFT Analytics
As organisations strive to meet the demands of the digital era, SIFT remains steadfast in its commitment to delivering transformative solutions. To explore digital transformation possibilities or learn more about SIFT’s pioneering work, contact the team for a complimentary consultation. Visit the website at www.sift-ag.com for additional information.
About SIFT Analytics
Get a glimpse into the future of business with SIFT Analytics, where smarter data analytics driven by smarter software solution is key. With our end-to-end solution framework backed by active intelligence, we strive towards providing clear, immediate and actionable insights for your organisation.
Headquartered in Singapore since 1999, with over 500 corporate clients, in the region, SIFT Analytics is your trusted partner in delivering reliable enterprise solutions, paired with best-of-breed technology throughout your business analytics journey. Together with our experienced teams, we will journey. Together with you to integrate and govern your data, to predict future outcomes and optimise decisions, and to achieve the next generation of efficiency and innovation.
The Analytics Times
“The Analytics Times is your source for the latest trends, insights, and breaking news in the world of data analytics. Stay informed with in-depth analysis, expert opinions, and the most up-to-date information shaping the future of analytics.
Published by SIFT Analytics
SIFT Marketing Team
marketing@sift-ag.com
+65 6295 0112
SIFT Analytics Group
Explore our latest insights
Artificial intelligence (AI) is expected to greatly improve industries like healthcare, manufacturing, and customer service, leading to higher-quality experiences for customers and employees alike. Indeed, AI technologies like machine learning (ML) have already helped data practitioners produce mathematical predictions, generate insights, and improve decision-making.
Furthermore, emerging AI technologies like generative AI (GenAI) can create strikingly realistic content that has the potential to enhance productivity in virtually every aspect of business. However, AI can’t succeed without good data, and this paper describes six principles for
ensuring your data is AI-ready
It would be foolish to believe that you could just throw data at various AI initiatives and expect magic to happen, but that’s what many practitioners do. While this approach might seem to work for the first few AI projects, data scientists increasingly spend more time correcting
and preparing the data as projects mature.
Additionally, data used for AI has to be high-quality and precisely prepared for these intelligent applications. This means spending many hours manually cleaning and enhancing the data to ensure accuracy and completeness, and organizing it in a way that machines can easily understand. Also, this data often requires extra information — like definitions and labels — to enrich semantic meaning for automated learning and to help AI perform tasks more effectively.
Therefore, the sooner data can be prepared for downstream AI processes, the greater the benefit. Using prepped, AI-ready data is like giving a chef pre-washed and chopped vegetables instead of a whole sack of groceries — it saves effort and time and helps ensure that the final dish is promptly delivered. The diagram below defines six critical principles for ensuring the “readiness” of data and its suitability for AI use.
The remaining sections of this paper discuss each principle in detail.
1.
Bias in AI systems, also known as machine learning or algorithm bias, occurs when AI applications produce results reflecting human biases, such as social inequality. This can happen when the algorithm development process includes prejudicial assumptions or, more commonly, when the training data has bias. For example, a credit score algorithm may deny a loan if it consistently uses a narrow band of financial attributes.
Consequently, our first principle focuses on providing a wide variety of data to AI models, which increases data diversity and reduces bias, helping to ensure that AI applications are less likely to make unfair decisions.
Diverse data means you don’t build your AI models on narrow and siloed datasets. Instead, you draw from a wide range of data sources spanning different patterns, perspectives, variations, and scenarios relevant to the problem domain. This data could be well-structured
and live in the cloud or on-premises. It could also exist on a mainframe, database, SAP system, or software as a service (SaaS) application. Conversely, the source data could be unstructured and live as files or documents on a corporate drive.
It’s essential to acquire diverse data in various forms for integration into your ML and GenAI applications.
2.
While it’s true that ML and GenAI applications thrive on diverse data, the freshness of that data is also crucial. Just as a weather forecast based on yesterday’s conditions isn’t conducive for a trip you plan to take today, AI models trained on outdated information can produce inaccurate or irrelevant results. Moreover, fresh data allows AI models to stay current with trends, adapt to changing circumstances, and deliver the best possible outcomes. Therefore, the second principle of AI-ready data is timeliness.
It’s critical that you build and deploy low-latency, real-time data pipelines for your AI initiatives to ensure timely data. Change data capture (CDC) is often used to deliver timely data from relational database systems, and stream capture is used for data originating from IoT devices that require low-latency processing. Once the data is captured, target repositories are updated and changes continuously applied in near-real time for the freshest possible data.
Remember, timely data enables more accurate and informed predictions.
3.
The success of any ML or GenAI initiative hinges on one key ingredient: correct data. This is because AI models act like sophisticated sponges that soak up information to learn and perform tasks. If the information is inaccurate, it’s like the sponge is soaking up dirty water, leading to
biased outputs, nonsensical creations, and, ultimately, a malfunctioning AI system. Therefore, data accuracy is the third principle and a fundamental tenet for building reliable and trustworthy AI applications.
Data accuracy has three aspects. The first is profiling source data to understand its characteristics, completeness, distribution, redundancy, and shape. Profiling is also commonly known as exploratory data analysis, or EDA.
The second aspect is operationalizing remediation strategies by building, deploying, and continually monitoring the efficacy of data quality rules. Your data stewards may need to be involved here to aid with data deduplication and merging. Alternatively, AI can help automate and accelerate the process through machine-recommended data quality suggestions.
The final aspect is enabling data lineage and impact analysis — with tools for data engineers and scientists that highlight the impact of potential data changes and trace the origin of data to prevent accidental modification of the data used by AI models.
High-quality, accurate data ensures that models can identify relevant patterns and
relationships, leading to more precise decisions, generation, and predictions.
4.
AI systems often use sensitive data — including personally identifiable information (PII), financial records, or proprietary business information — and use of this data requires responsibility. Leaving data unsecured in AI applications is like leaving a vault door wide open. Malicious actors could steal sensitive information, manipulate training data to bias outcomes, or even disrupt entire GenAI systems. Securing data is paramount to protecting privacy, maintaining model integrity, and ensuring the responsible development of powerful AI applications. Therefore, data security is the fourth AI-ready principle.
Again, three tactics can help you automate data security at scale, since it’s nearly impossible to do it manually. Data classification detects, categorizes, and labels data that feeds the next stage. Data protection defines policies like masking, tokenization, and encryption to obfuscate the data. Finally, data security defines policies that describe access control, i.e., who can access the data. The three concepts work together as follows: first, privacy tiers should be defined and data tagged with a security designation of sensitive, confidential, or restricted. Next, a protection policy should be applied to mask restricted data. Finally, an access control policy should be used to limit access rights.
These three tactics protect your data and are crucial for improving
the overall trust in your AI system and safeguarding its reputational value.
5.
The principles we’ve discussed so far have primarily focused on promptly delivering the right data, in the correct format, to the right people, systems, or AI applications. But stockpiling data isn’t enough. AI-ready data has to be discoverable and readily accessible within the system. Imagine a library with all the books locked away — the knowledge is there but unusable. Discoverable data unlocks the true potential of ML and GenAI, allowing these workloads to find the information they need to learn, adapt, and produce ground-breaking results. Therefore,
discoverability is the fifth principle of AI-ready data.
Unsurprisingly, good metadata practices lie at the center of discoverability. Aside from the technical metadata associated with AI datasets, business metadata and semantic typing must also be defined. Semantic typing provides extra meaning for automated systems, while additional business terms deliver extra context to aid human understanding. A best practice is to create a business glossary that maps business terms to technical items in the datasets, ensuring a common understanding of concepts. AI-assisted augmentation can also be used to automatically generate documentation and add business semantics from the glossary. Finally, all the metadata is indexed and made searchable via a data catalog.
This approach ensures that the data is directly discoverable, applicable,
practical, and significant to the AI task at hand.
6.
We’ve already mentioned that ML and GenAI applications are mighty tools, but their potential rests on the ability to readily consume data. Unlike humans, who can decipher handwritten notes or navigate messy spreadsheets, these technologies require information to be represented in specific formats. Imagine feeding a picky eater — if they won’t eat what you’re serving, they’ll go hungry. Similarly, AI initiatives won’t be successful if the data is not in the right format for ML experiments or LLM applications. Making data easily consumable helps unlock the potential of these AI systems, allowing them to ingest information smoothly and translate it into intelligent actions for creative outputs. Consequently, making data readily consumable is the final principle of AI-ready data.
Making Data Consumable for Machine Learning
Data transformation is the unsung hero of consumable data for ML. While algorithms like linear regression grab the spotlight, the quality and shape of the data they’re trained on are just as critical. Moreover, the effort invested in cleaning, organizing, and making data consumable by ML models reaps significant rewards. Prepared data empowers models to learn effectively, leading to accurate predictions, reliable outputs, and, ultimately, the success of the entire ML project.
However, the training data formats depend highly on the underlying ML infrastructure. Traditional ML systems are disk-based, and much of the data scientist workflow focuses on establishing best practices and manual coding procedures for handling large volumes of files. More recently, lakehouse-based ML systems have used a database-like feature store, and the data scientist workflow has transitioned to SQL as a first-class language. As a result, well-formed, high-quality, tabular data structures are the most consumable and convenient data format
for ML systems.
Making Data Consumable for Generative AI
Large language models (LLMs) like OpenAI’s GPT-4, Anthropic’s Claude, and Google AI’s LaMDA and Gemini have been pre-trained on masses of text data and lie at the heart of GenAI. OpenAI’s GPT-3 model was estimated to be trained with approximately 45 TB of
data, exceeding 300 billion tokens. Despite this wealth of inputs, LLMs can’t answer specific questions about your business, because they don’t have access to your company’s data. The solution is to augment these models with your own information, resulting in more correct, relevant, and trustworthy AI applications.
The method for integrating your corporate data into an LLM-based application is called retrieval augmented generation, or RAG. The technique generally uses text information derived from unstructured, file-based sources such as presentations, mail archives, text documents, PDFs, transcripts, etc. The text is then split into manageable chunks and converted into a numerical representation used by the LLM in a process known as embedding. These embeddings are then stored in a vector database like Chroma, Pinecone, and Weviate. Interestingly, many traditional database vendors — such as PostgreSQL, Redis, and SingleStoreDB — also support vectors. Moreover, cloud platforms like Databricks, Snowflake, and Google BigQuery have recently added support for vectors, too.
Whether your source data is structured or unstructured, Qlik’s approach ensures that
quality data is readily consumable for your GenAI, RAG, or LLM-based applications.
Having defined the six core principles of data readiness and suitability, the questions remain: can the principles be codified and easily translated for everyday use? And how can the readiness for AI be quickly discerned? One possibility is to use Qlik’s AI Trust Score as a global and understandable readiness indicator.
The AI Trust Score assigns a separate dimension for each principle and then aggregates each value to create a composite score, a quick, reliable shortcut to assessing your data’s AI readiness. Additionally, because enterprise data continually changes, the trust score is regularly checked and frequently recalibrated to track data readiness trends.
Your AI Trust Score aggregates multiple metrics into a single, easy to-understand readiness Score.
The need for high-quality, real-time data that drives more thoughtful decisions, operational efficiency, and business innovation has never been greater. That’s why successful organizations seek market-leading data integration and quality solutions from Qlik Talend to efficiently deliver trusted data to warehouses, lakes, and other enterprise data platforms. Our comprehensive, best-in-class offerings use automated pipelines, intelligent transformations, and reliable Datasets quality to provide the agility data professionals crave with the governance and
compliance organizations expect.
So, whether you’re creating warehouses or lakes for insightful analytics, modernizing operational data infrastructures for business efficiency, or using multi-cloud data for artificial intelligence initiatives, Qlik Talend can show you the way.
Despite machine learning’s transformative power and generative AI’s explosive growth potential, data readiness is still the cornerstone
of any successful AI implementation. This paper described six key principles for establishing a robust and trusted data foundation that
combine to help your organization unlock AI’s true potential.
Next Steps
For more information or enquiries about Qlik products and services, feel free to contact us below.
Connect with SIFT Analytics
As organisations strive to meet the demands of the digital era, SIFT remains steadfast in its commitment to delivering transformative solutions. To explore digital transformation possibilities or learn more about SIFT’s pioneering work, contact the team for a complimentary consultation. Visit the website at www.sift-ag.com for additional information.
About SIFT Analytics
Get a glimpse into the future of business with SIFT Analytics, where smarter data analytics driven by smarter software solution is key. With our end-to-end solution framework backed by active intelligence, we strive towards providing clear, immediate and actionable insights for your organisation.
Headquartered in Singapore since 1999, with over 500 corporate clients, in the region, SIFT Analytics is your trusted partner in delivering reliable enterprise solutions, paired with best-of-breed technology throughout your business analytics journey. Together with our experienced teams, we will journey. Together with you to integrate and govern your data, to predict future outcomes and optimise decisions, and to achieve the next generation of efficiency and innovation.
The Analytics Times
“The Analytics Times is your source for the latest trends, insights, and breaking news in the world of data analytics. Stay informed with in-depth analysis, expert opinions, and the most up-to-date information shaping the future of analytics.
Published by SIFT Analytics
SIFT Marketing Team
marketing@sift-ag.com
+65 6295 0112
SIFT Analytics Group
Explore our latest insights
Unlock your SAP data for faster time to insight and value
Today’s digital environment demands instant access to information for real-time decision-making, improved business agility, amazing customer service, and competitive advantage. Every person in your business, no matter what their role, requires easy access to the most accurate data set to make informed decisions – especially when it comes to SAP.
Industries like Manufacturing, Retail, CPG, Oil and Gas, and many others rely on actionable data from SAP to make strategic decisions and maintain competitive advantage
• Reduce time to insight with increased speed and agility
• More self-service analytics
• Use SAP data anywhere
• Utilize 3rd party software tools most
suited for the job at hand
• Combine SAP + non-SAP data
• Real-time, predictive (AI/ML)
• Facilitate scale with better cost control
• Reduce SAP Analytics TCO
So why doesn’t every company embrace the opportunity to use new analytic platforms — by streaming live SAP data for real time analytics or combining it with other data sources in data lake and data warehouse platforms in the cloud?
SAP systems come with a number of unique challenges that are inherent to the platform. They are challenging to integrate because they are structurally complex with tens of thousands of tables that have intricate relationships as well as proprietary data formats making data inaccessible outside of SAP applications.
This complexity means that integrating data for analytics can be cumbersome, time consuming and costly.
SAP datasets are full of value, but they won’t do your organization any good unless you can easily use them in a cost-effective, secure way
Realize the value of all your SAP data with Qlik® and Snowflake
Qlik helps you efficiently capture large volumes of changed data from your source SAP systems, as well as other enterprise data sources, and deliver analytics ready data in real-time to Snowflake. Data sets are then easily cataloged, provisioned, and secured for all your analytic initiatives such as AI, machine learning, data science, BI
and operational reporting.
Qlik and Snowflake remove the business and technical challenges of complexity, time and cost, while gaining flexibility and agility for your SAP data. This unlocks your SAP data for different analytics use cases, and allows you to combine it with other enterprise data for even deeper insights and increased value.
SOLUTION USE CASE
Data Warehousing
Quickly load a new SAP data into the Snowflake Data Cloud in real-time gaining new and valuable insights.
Data Lake
Easily ingest or migrate SAP data into the Snowflake Data Cloud to support a variety of workloads unifying and securing all your organizations data in one place.
Data Science & AI/ML
Speed up workflows and transform SAP data into ML-powered insights using your language of choice with Snowflake.
The Snowflake platform provides a simple and elastic alternative for SAP customers that simultaneously ensures an organization’s critical information is protected. Snowflake was built from scratch to run in the cloud and frees companies from cloud provider lock-in.
Architectural Simplicity
Snowflake improves the ease of use of SAP with greater architectural simplicity. Snowflake can ingest data from SAP operations systems (both on-premises and cloud), third-party systems, and signal data – whether its in structured or semi-structured formats. This provides a foundation for customers to build 360-degree views of customers, products, and the supply chain enabling trusted business content to be accessible to all users.
Convenient Workload Elasticity
Snowflake’s multi-cluster shared data architecture separates compute from storage, enabling customers to elastically scale, up and down, automatically or on the fly. Users can apply dedicated compute clusters to each workload in near-unlimited quantities for virtually unlimited concurrency without contention. Once
a workload is completed, compute is dialed back down in seconds so customers don’t have to deal with the frustrations of throttling SAP BW queries.
Reliable Data Security
With Snowflake, simplicity also means data security. All data is always encrypted—in storage or in transit—as a built-in feature of the platform. Data is landed
once and views are shared out. This means one copy, one security model, and hundreds of elastic compute clusters with monitors on each one of them.
Breaking the barriers to modernizing SAP data analytics
Ingest and deliver SAP data to Snowflake in real time.
Qlik accelerates the availability of SAP data to Snowflake in real-time with its scalable change data capture technology. Qlik Data Integration supports all core SAP and industry modules, including ECC, BW, CRM, GTS, and MDG, and continuously delivers incremental data updates with metadata to Snowflake in real time.
Automate the data warehouse lifecycle and pipeline.
Once your SAP data has landed in the Snowflake Data Cloud, Qlik automates the data pipeline without the hand coding associated with ETL approaches. You can efficiently model, refine and automate data warehouse or lake lifecycles to increase agility and productivity.
SAP Solution Accelerators
Qlik also provides a variety of Business Solutions such as Order to Cash. Unique SAP accelerators with preconfigured integrations and process flows leverage Qlik’s deep technical mastery of SAP structures and complexities. They enable real-time ingestion (CDC) and rapid integration of SAP data into Snowflake. The automated mapping and data model generation eliminate the need for expensive, highly technical and risky manual mapping and migration processes.
Discover bolder business insights.
Use business analytics tools such as Qlik Sense or any other BI tool to explore the data, create interactive dashboards, and carry out a variety of BI use cases for data-driven insights.
Real-time data ingestion (CDC) from SAP to Snowflake
Automated mapping and data model generation for analytics (data marts)
Prepackaged solution accelerators for common business use cases (order-to-cash, financials,
inventory management)
Decode SAP proprietary source structures (pool/cluster tables, HANA/CDS views)
Supports all core and industry-specific SAP modules
World-class SAP expertise to support presales and customer success
Qlik is a Snowflake Elite partner with Snowflake Ready validated Solutions for Data Integration and Analytics. We accelerate time-to-insight with our end-to-end data integration and analytics solution taking you from your unrealized data value in your SAP landscape to informed action in your Snowflake Data Cloud. Qlik’s solution for Snowflake users automates the design, implementation and updates of data models while minimizing the manual, error-prone design processes of data modeling, ETL coding and scripting. As a result, you can accelerate
analytics projects, achieve greater agility and reduce risk – all while fully realizing the instant elasticity and cost advantages of Snowflake’s Data Cloud.
Next Steps
For more information or enquiries about Qlik products and services, feel free to contact us below.
Connect with SIFT Analytics
As organisations strive to meet the demands of the digital era, SIFT remains steadfast in its commitment to delivering transformative solutions. To explore digital transformation possibilities or learn more about SIFT’s pioneering work, contact the team for a complimentary consultation. Visit the website at www.sift-ag.com for additional information.
About SIFT Analytics
Get a glimpse into the future of business with SIFT Analytics, where smarter data analytics driven by smarter software solution is key. With our end-to-end solution framework backed by active intelligence, we strive towards providing clear, immediate and actionable insights for your organisation.
Headquartered in Singapore since 1999, with over 500 corporate clients, in the region, SIFT Analytics is your trusted partner in delivering reliable enterprise solutions, paired with best-of-breed technology throughout your business analytics journey. Together with our experienced teams, we will journey. Together with you to integrate and govern your data, to predict future outcomes and optimise decisions, and to achieve the next generation of efficiency and innovation.
The Analytics Times
“The Analytics Times is your source for the latest trends, insights, and breaking news in the world of data analytics. Stay informed with in-depth analysis, expert opinions, and the most up-to-date information shaping the future of analytics.
Published by SIFT Analytics
SIFT Marketing Team
marketing@sift-ag.com
+65 6295 0112
SIFT Analytics Group
Explore our latest insights
A long-time leader and innovator in the data, analytics, and AI space, Qlik is perfectly positioned to fully embrace AI — not only in our products, but also in the way we conduct business — and do so responsibly. As the rise of generative AI accelerated the requirement for organizations to modernize their data fabric, it created new opportunities for Qlik to innovate in support of our customers’ efforts in developing and implementing their AI strategies. Over the past year, we have continued to lead through new acquisitions, product
innovation, talent development, technology investments, and by establishing new systems and processes.
Pillar 1
AI can’t succeed without good data: It is fully dependent on an organization’s ability to establish a trusted data foundation. This was already the case with predictive AI, but the rise of generative AI — which relies on data to function — has accelerated the need for companies to modernize their data fabric. Our point of view is that there are six principles to follow for creating AI-ready data and our
product strategy for our data integration and quality portfolio fully aligns to them:
1. Data should be diverse (coming from a wide range of resources) to remove bias in AI systems
2. Data should be timely to make accurate and informed predictions
3. Data should be accurate to ensure reliability and trustworthiness in AI
4. Data should be secure to safeguard the reputation of your AI
5. Data should be discoverable to enable use of relevant and contextual data
6. Data should be consumable for ML training and LLM integration
Our Portfolio
Our data integration portfolio has always been designed to move data from any source to any target, in real time. As these destinations will often use AI on this data, this is data integration operating in the service of AI, including generative AI. Qlik’s differentiation is our ability to take the best-in-class capabilities that we are known for (real-time data integration and transformation at scale) and make
them available for generative AI use cases.
In July 2024, we launched Qlik Talend Cloud®. This new flagship offering combines the best functionality of legacy solutions Qlik Cloud® Data Integration, Talend® Cloud, and Stitch Data, and is designed to help our customers implement a trusted data foundation for AI.
Qlik Talend Cloud is built on Qlik’s cloud infrastructure platform, with the focus on managing the data integrity of our customers’ AI, analytics, and business operational projects. It offers a unified package of data integration and quality capabilities that enable data engineers and scientists to deploy AI-augmented data pipelines that deliver trusted data wherever it’s needed. This includes:
We have provided productivity-enhancing tools (like a co-pilot) for data engineers (prompt to SQL), with more coming later this year.
What’s Next
For the latter part of 2024, we plan to introduce a range of dedicated components to support RAG implementations with the leading vector databases, embedding models, and LLMs. This will offer data engineers implementing AI workloads the same reliability and scalability they expect when operationalizing all their other workloads.
Looking ahead, our 2025 plan includes further enhancements through generative AI to further improve data engineer productivity, including data pipeline design tasks, dataset auto-classifications, automated workflows, and AI-assisted record deduplication.
WHO’S IT FOR
Data Engineers and Data Architects
These professionals need to ensure that data that will be used for downstream AI processes is of high quality and trustworthy. They also want to be able to deliver that data throughout their organization using AI-augmented, no-code pipelines.
Pillar 2
Enriching analytical applications and workflows with AI-powered capabilities promotes enhanced, data-centric decision making and accelerates insights. While there has been much hype around generative AI over the last year, our point of view is that it isn’t the solution to everything. Instead, we believe that both predictive AI (i.e. traditional AI), which processes and returns expected results such as analyses and predictions, and generative AI, which produces newly synthesized content based on training from existing data, hold huge potential.
Therefore, our product strategy for our analytics portfolio encompasses both predictive and generative AI.
Our Portfolio
AI has always been foundational to Qlik Cloud Analytics, our flagship analytics offering. From analytics creation and data prep to data exploration — with natural language search, conversational analytics, and natural language generation — Qlik Cloud® is designed to enhance everything users do with AI.
Today, we offer a full range of AI-powered, augmented analytics capabilities that deepen insight, broaden access, and drive efficiency. This includes:
What’s Next
Our product roadmap for Analytics AI is about enhancing outcomes through automation and integrated intelligence, spanning the following tenets of AI-powered analytics:
WHO’S IT FOR
Application Creators and Users
These professionals are looking to build and use AI-infused applications for more powerful data analysis to support decision making — and do it in a way that is intelligent, automated, embedded, and intuitive (hence easier to adopt).
Pillar 3
Companies today are looking to create value with AI by building and deploying AI models. But following the hype of generative AI in 2023, this year there has been a shift in focus1 from large language models, which necessitate significant investments, to smaller models that are more cost efficient, easier, and faster to build and deploy.
Qlik’s product strategy is perfectly aligned to this shift. We offer self-service AI solutions that enable companies to deliver an AI experience for advanced, specific use cases in a way that is efficient and affordable with fast time to value.
Our Portfolio
In July 2024, we launched Qlik Answers™, a plug and-play, generative AI-powered knowledge assistant. Qlik Answers is a self-service AI solution that can operate independently from other Qlik products and is sold separately.
This tool allows organizations to deploy an AI model that can deliver answers from a variety of unstructured data sources. The ability to analyze unstructured data enables Qlik to deliver unique value to our customers, as it’s commonly believed that 80% of the world’s data is unstructured2. A study that the firm ETR conducted on our behalf in April 2024 also found that while companies understood the value potential of being able to deliver insights from unstructured data, less than one-third felt their organization was well equipped to do so.
With Qlik Answers, organizations can now take advantage of an out-of-the-box, consolidated self-service solution that allows users to get personalized, relevant answers to their questions in real time with full visibility of source materials. As with all Qlik products, our customers can also be assured that their data stays private. Moreover, with Qlik Answers, users will only have access to data that is curated for a specific
use case. With multiple, domain-specific knowledge bases being accessible to assistants, organizations stay in control of what content users can access.
To help ensure a successful implementation, our pricing and packaging for Qlik Answers includes starter services delivered by our customer success organization.
Since 2021, Qlik has been offering another self-service AI solution for predictive analytics, Qlik AutoML®. Like Qlik Answers, Qlik AutoML can
be purchased separately.
Qlik AutoML provides a guided, no-code machine learning experience that empowers analytics teams to perform predictive analytics without the support of data science teams. With AutoML, users can:
Note: While AutoML runs inside of Qlik Cloud, it can also be used independent of Qlik Cloud Analytics. We have customers who use a real-time API to return predictions back to their own systems without having to access Qlik Cloud.
Finally, Qlik also offers connectors to enable its customers to integrate third-party generative AI models in their analytics apps, load scripts, and automations. Qlik Cloud customers have the option to leverage our AI Accelerator program to integrate large language models into their applications.
What’s Next
In September 2024, we introduced new enhancements to Qlik AutoML’s capabilities,
including augmented MLOps, model optimization, and analytics views, with plans
for additional upgrades through the end of the year and into 2025. Future improvements are focused on the ability to create time-aware models and the introduction of a full, end-to-end MLOps lifecycle for models developed on the platform to ensure they can be adequately monitored and governed.
Although Qlik Answers is a new product, we’ve already augmented its knowledge base and assistant capabilities, with more enhancements planned.
WHO’S IT FOR
Decision-Makers and End Users
These professionals want to leverage AI in a self-service way to get insights and answers that will help them make the best predictions and decisions for their area(s) of responsibility.
In order to continue to develop innovative AI products and capabilities — and to ensure we do so with ethical integrity — we have put in place a rich ecosystem of AI expertise to help steer our strategy and direction. Above all, we are deeply committed to the responsible development and deployment of our technology in ways that earn and maintain people’s trust.
Principles for Responsible AI
We have created a set of principles guiding the responsible development and deployment of our technology, available publicly at qlik.com/Trust/AI. These principles are:
Reliability: We design our products for high performance and availability so customers can safely and securely integrate and analyze data and use it to make informed decisions.
Customer control: We believe customers should always remain in control of their data and how their data is used so we design our products with finegrain security controls, including down to the row (data) and object level.
Transparency and explainability: We design our products to make it clear when customers engage with AI. We strive to make clear the data, analysis, limitations, and/or model used to generate AI-driven answers so our customers can make informed decisions on how they use our technology
Observability: We design our products so customers can understand lineage, access, and governance of data, analytics, and AI models used to inform answers and automate tasks.
Inclusive: We believe diversity, equity, inclusion, and belonging drive innovation and will continue to foster these beliefs through our product design and development.
Qlik has a process and staff in place to monitor for any upcoming legislation that would impact our business, such as new AI laws. As legislative changes occur, we assess these laws and adjust our AI compliance program accordingly.
Next Steps
For more information or enquiries about Qlik products and services, feel free to contact us below.
Connect with SIFT Analytics
As organisations strive to meet the demands of the digital era, SIFT remains steadfast in its commitment to delivering transformative solutions. To explore digital transformation possibilities or learn more about SIFT’s pioneering work, contact the team for a complimentary consultation. Visit the website at www.sift-ag.com for additional information.
About SIFT Analytics
Get a glimpse into the future of business with SIFT Analytics, where smarter data analytics driven by smarter software solution is key. With our end-to-end solution framework backed by active intelligence, we strive towards providing clear, immediate and actionable insights for your organisation.
Headquartered in Singapore since 1999, with over 500 corporate clients, in the region, SIFT Analytics is your trusted partner in delivering reliable enterprise solutions, paired with best-of-breed technology throughout your business analytics journey. Together with our experienced teams, we will journey. Together with you to integrate and govern your data, to predict future outcomes and optimise decisions, and to achieve the next generation of efficiency and innovation.
The Analytics Times
“The Analytics Times is your source for the latest trends, insights, and breaking news in the world of data analytics. Stay informed with in-depth analysis, expert opinions, and the most up-to-date information shaping the future of analytics.
Published by SIFT Analytics
SIFT Marketing Team
marketing@sift-ag.com
+65 6295 0112
SIFT Analytics Group
Explore our latest insights
“Everybody’s in the same place. They can talk to each other and see the same information on different dashboards updated in near real time. That’s the kind of interaction Qlik is enabling.”
Jordana Reis, Enterprise Integration Architect, Vale S.A.
Customer Name
Vale S.A.
Industry
Mining
Geography
Brazil
Function
Sales, Supply Chain Management
Business Value Driver
New Business Opportunities, Reimagined Processes
Challenges
Solution
Using Qlik Data Integration to handle and automate ETL processes, Vale developed the Integrated Operations Center to provide a clear overview of the supply chain.
Results
Brazil’s primary economic sector comprises critical industries such as agriculture, forestry and mining, all of which act as key sources of food, fuel and raw materials. Business units range in size from subsistence smallholdings to global giants with worldwide operations. And at the apex of the mining industry sits Vale S.A.
Founded 80 years ago, the Brazil-based metals and mining corporation is the world’s largest producer of iron ore and nickel. Vale is also the most valuable business in Latin America, with an estimated market value of $111 billion and rising, and a presence in 30 countries.
While mining remains the core of its business, Vale’s operations also encompass logistics, including an extensive network of railroads, ports and blending terminals, and shipping which distributes the company’s products across the world. Also supporting its operations are Vale’s own power plants and iron pelletizing facilities.
Vale’s dry bulk supply chain is also a large-scale service, and one of the biggest transport and distribution operations in Brazil. Vale owns around 490 locomotives and more than 29,500 rail freight cars, and ships much of its iron ore and pellet output from Brazil, around the African coast, to China and Malaysia, often in its own or chartered vessels, including Very Large Ore Carriers (VLOCs).
Managing Vale’s global operation involves a series of complex and resource-intensive distribution processes. These were placed into sharp focus in 2015 when the business faced falling commodities prices and an increasingly competitive market.
“The geographic distances we cover, from the extraction of iron ore to delivery to customers, are very long,” says Jordana Reis, Enterprise Integration Architect at Vale. “That becomes an even bigger issue when our main competitors are closer to our buyers than we are.”
Vale’s operations were managed by a series of manual and largely disconnected processes, with different departments handling their own functions and using their own methodologies, often with legacy systems. “There were people looking at the mining aspect, people looking at ports, people looking at sales, but we didn’t have an integrated view of these operations,” explains
Richardson Nascimento, Data and AI Architect at Vale. “That was the process we needed to fix.”
This lack of an integrated view of the business was causing a range of challenges, including mismatches between production and transport capacity, logistical inefficiency and product quality management issues. “We were also missing out on valuable sales opportunities, simply because we didn’t know if we could fulfill them,” recalls Reis.
Vale developed the Centro de Operações Integradas (Integrated Operations Center, or COI) as an operating model. One of its pillars is to provide a means of aggregating and processing the vast amounts of data it was generating but only partially using. The COI would then act as a central framework, updated in near real time, on which Vale could base decisions, better manage its production and supply chain and support its people and processes.
“When we realized how much data we would need to move to really enable COI, we started thinking about how we could automate the process,” says Nascimento. “The main driver was low latency replication. We had a target to move all this information in less than 15 minutes, and Qlik Data Integration was clearly the best option.”
Vale collaborated closely with both Microsoft and Qlik teams during the purchase process. “Both teams were very active and interested in making COI happen,” says Reis. “They gave us honest opinions and helped us to achieve our goals.”
COI uses Qlik Replicate IaaS with Microsoft Azure in tandem with a range of data repositories such as Azure SQL Database and Azure Synapse, with Qlik Replicate acting as the principal enabler of the process. Another key factor in the choice of Qlik Data Integration was agentless operation, and its efficiency in reading application databases and transaction logs without impacting their activity.
COI’s main data sources are Vale’s in-house Manufacturing Execution Systems (MES), responsible for each stage of the value chain (Mining, Rail, Ports and Pelletizing), all based on Oracle databases; the chartering system Softmar and VesselOps,
based on SQL Server; and Vale’s in-house value chain optimization systems, also based on Oracle databases.
Nascimento also points to Qlik Data Integration’s importance in supporting tools such as Azure Databricks as part of Vale’s strategy to use machine learning and artificial intelligence to augment human decisions. Vale is using several tools for big data processing, such as Azure Machine Learning. “That’s one of the tools that we’re trying to leverage more,” he notes. “Azure Machine Learning is simple to use and easy to teach.”
Importantly, Reis highlights Qlik’s ease of use and speed of implementation and operation. “It changed our extract, transform and load (ETL) process and how we make data available,” she notes. “We reduced the effort to make data available to build less complex dashboards, for instance, from four weeks to just four hours.”
COI began to deliver benefits almost immediately on its launch in 2017. It enabled a new integrated planning process, giving staff across the business full visibility into the supply chain improving the ability to manage their respective operations
in a collaborative environment.
“Everything related to operations is now under COI’s umbrella,” says Nascimento. “It covers the mines, the ports, railroads, shipping and sales and freight negotiations. COI enables planning and optimization across the supply chain.”
Users can now define and build their own dashboards, while corporate dashboards also enable insights and support decisions at board level. COI’s value is neatly encapsulated in Vale’s videowalls, giant room-sized panels featuring custom dashboards that enable cross-functional collaboration. “Everybody’s in the same place,” says Reis.
“They can talk to each other and see the same information on different dashboards updated in near real time. That’s the kind of interaction Qlik is enabling.”
Nascimento also highlights Vale’s asset monitoring center, which uses a similar and connected operating model to COI that combines with other tools to provide insights into asset lifecycles, enabling preventive maintenance and extending the efficiency and working lives of machinery, plant, vehicles and more.
“It’s not just about the speed of the decisions, but that we can make different types of decisions,” Nascimento explains. “We can now adjust production in line with logistical capacities, for example. And that’s transformational.”
The initial launch of COI in 2017 delivered staggering results almost immediately, enabling business benefits in terms of sales won, costs saved and efficiencies gained totaling $300 million after just one month of operation and $600 million annual savings.
This, however, is just the start. COI is what Reis describes as “a lighthouse project”, with the data architecture implemented by the Integrated Operations Center and enabled by Qlik now used across multiple other projects covering areas such as safety,
geotechnical methods and autonomous machinery.
“Our long-term strategy is based on Qlik and Microsoft Azure. Once we saw the benefits on COI, we set Qlik Data Integration as our target information integration architecture for the whole enterprise,” concludes Reis. “We also have a program to migrate as many systems as possible to Microsoft Azure, including our data repositories for analytics. And of course, we will use Qlik Data Integration and Qlik Compose there too.”
Next Steps
For more information or enquiries about Qlik products and services, feel free to contact us below.
Connect with SIFT Analytics
As organisations strive to meet the demands of the digital era, SIFT remains steadfast in its commitment to delivering transformative solutions. To explore digital transformation possibilities or learn more about SIFT’s pioneering work, contact the team for a complimentary consultation. Visit the website at www.sift-ag.com for additional information.
About SIFT Analytics
Get a glimpse into the future of business with SIFT Analytics, where smarter data analytics driven by smarter software solution is key. With our end-to-end solution framework backed by active intelligence, we strive towards providing clear, immediate and actionable insights for your organisation.
Headquartered in Singapore since 1999, with over 500 corporate clients, in the region, SIFT Analytics is your trusted partner in delivering reliable enterprise solutions, paired with best-of-breed technology throughout your business analytics journey. Together with our experienced teams, we will journey. Together with you to integrate and govern your data, to predict future outcomes and optimise decisions, and to achieve the next generation of efficiency and innovation.
The Analytics Times
“The Analytics Times is your source for the latest trends, insights, and breaking news in the world of data analytics. Stay informed with in-depth analysis, expert opinions, and the most up-to-date information shaping the future of analytics.
Published by SIFT Analytics
SIFT Marketing Team
marketing@sift-ag.com
+65 6295 0112
SIFT Analytics Group
Explore our latest insights
Being able to analyze high-volume, varied datasets is essential in nearly all industries. From fraud detection and real-time customer offers to market trend and pricing analysis, analytics use cases are boosting competitive advantage. In addition, the advent of the Internet of Things (IoT) and Artificial Intelligence (AI) are also driving up the volume and variety of data that organizations like yours want and need to analyze. The challenge: as the speed of business accelerates, data has increasingly perishable value. The solution: real-time data analysis.
Data lakes have emerged as an efficient and scalable platform for IT organizations to harness all types of data and enable analytics for data
scientists, analysts, and decision makers. But challenges remain. It’s been too hard to realize the expected returns on data lake investments, due to several key challenges in the data integration process ranging from traditional processes that are unable to adapt to changing platforms and data transfer bottlenecks to cumbersome manual scripting, lack of scalability, and the inability to quickly and easily extract source data.
Qlik®, which includes the former Attunity data integration portfolio, helps your enterprise overcome these obstacles with fully automated, high-performance, scalable, and universal data integration software.
Combining efficient distributed processing with cost-effective storage for mixed data sets analysis forever redefined the economics and possibilities of analytics. Data lakes were initially built on three pillars: the Hadoop foundation of MapReduce batch processing, the Hadoop Distributed File System (HDFS), and a “schema on read” approach that does not structure data until it’s analyzed. These pillars are evolving:
While the pillars are evolving, the fundamental premise of the data lake remains the same:
organizations can benefit from collecting, managing, and analyzing multi-sourced data on distributed commodity storage and processing resources.
As deployments proceed at enterprises across the globe, IT departments face consistent challenges when it comes to data integration. According to the TDWI survey (Data Lakes: Purposes, Practices, Patterns and Platforms), close to one third (32%) of respondents were concerned about their lack of data integration tools and related Hadoop programming skills.
Traditional data integration software tools are challenging, too, because they were designed last century for databases and data warehouses. They weren’t architected to meet the high-volume, real-time ingestion requirements of data lake, streaming, and cloud platforms. Many of these tools also use intrusive replication methods to capture transactional data, impacting production source workloads.
Often, these limitations lead to rollouts being delayed and analysts forced to work with stale and/or insufficient datasets. Organizations struggle to realize a return on their data lake investment. Join the most successful IT organizations in addressing these common data lake challenges by adopting the following five core architectural principles.
1. Plan on Changing Plans
Your architecture, which likely will include more than one data lake, must adapt to changing requirements. For example, a data lake might start out on premises and then be moved to the cloud or a hybrid environment. Alternatively, the data lake might need to run on Amazon Web Services, Microsoft Azure, or Google platforms to complement on-premises components.
To best handle constantly changing architectural options, you and your IT staff need platform flexibility. You need to be able to change sources and targets without a major retrofit of replication processes.
Qlik Replicate™ (formerly Attunity Replicate) meets these requirements with a 100% automated process for ingesting data from any major source (e.g., database, data warehouse, legacy/mainframe, etc.) into any major data lake based on HDFS or S3. Your DBAs and data architects can easily configure, manage, and monitor bulk or real-time data flows across all these environments.
You and your team also can publish live database transactions to messaging platforms such as Kafka, which often serves as a channel to data lakes and other Big Data targets. Whatever your source or target, our Qlik Replicate solution provides the same drag-and-drop configuration
process for data movement, with no need for ETL programming expertise.
Two Potential Data Pipelines — One CDC Solution
2. Architect for Data in Motion
For data lakes to support real- time analytics, your data ingestion capability must be designed to recognize different data types and multiple service-level agreements (SLAs). Some data might only require batch or microbatch processing, while other data requires stream processing tools or frameworks (i.e., to analyze data in motion). To support the complete range, your system must be designed to support technologies such as Apache Kafka, Amazon Kinesis, Azure Event Hubs, and Google Cloud Pub/Sub as needed.
Additionally, you’ll need a system that ensures all replicated data can be moved securely, especially when sensitive data is being moved to a cloud-based data lake. Robust encryption and security controls are critical to meet regulatory compliance, company policy, and end-user
security requirements.
Qlik Replicate CDC technology non-disruptively copies source transactions and sends them at near-zero latency to any of the real- time/messaging platforms listed above. Using log reader technology, it copies source updates from database transaction logs – minimizing impact on production workloads – and publishes them as a continuous message stream. Source DDL/schema changes are injected into this stream to ensure analytics workloads are fully aligned with source structures. Authorized people also can transfer data securely and at high speed across the wide-area network (WAN) to cloud-based data lakes, leveraging AES-256 encryption and dynamic multipathing.
As an example, a US private equity and venture capital firm built a data lake to consolidate and analyze operational metrics from its portfolio companies. This firm opted to host its data lake in the Microsoft Azure cloud rather than taking on the administrative burden of an on-premises infrastructure. Qlik Replicate CDC captures updates and DDL changes from source databases (Oracle, SQL Server, MySQL, and DB2) at four locations in the US. Qlik Replicate then sends that data through an encrypted File Channel connection over a WAN to a virtual machine–based instance of Qlik Replicate in Azure cloud.
This Qlik Replicate instance publishes the data updates to a Kafka message broker that relays those messages in the JSON format to Spark. The Spark platform prepares the data in microbatches to be consumed by the HDInsight data lake, SQL data warehouse, and various other
internal and external subscribers. These targets subscribe to topics that are categorized by source tables. With the CDC-based architecture, this firm is now efficiently supporting real-time analysis without affecting production operations.
3. Architect for Data in Motion
Your data lake runs the risk of becoming a muddy swamp if there is no easy way for your users to access and analyze its contents. Applying technologies like Hive on top of Hadoop helps to provide an SQL-like query language supported by virtually all analytics tools. Organizations like yours often need both an operational data store (ODS) for up-to-date business intelligence (BI) and reporting as well as a comprehensive historical data store (HDS) for advanced analytics. This requires thinking about the best approach to building and managing these stores to deliver the agility the business needs.
This is more easily said than done. Once data is ingested and landed in Hadoop, often IT still struggles to create usable analytics data stores. Traditional methods require Hadoop-savvy ETL programmers to manually code the various steps – including data transformation, the creation of Hive SQL structures, and reconciliation of data insertions, updates, and deletions to avoid locking and disrupting users. The administrative burden of ensuring data is accurate and consistent can delay and even kill analytics projects.
Qlik Compose™ for Data Lakes (formerly Attunity Compose for Data Lakes) solves these problems by automating the creation and loading of Hadoop data structures, as well as updating and transforming enterprise data within the data store. You, your architects, or DBAs can automate the pipeline of BI ready data into Hadoop, creating both an ODS and HDS. Because our solution leverages the latest innovations in Hadoop such as the new ACID Merge SQL capabilities, available today in Apache Hive you can automatically and efficiently process data insertions, updates, and deletions. Qlik Replicate integrates with Qlik Compose for Data Lakes to simplify and accelerate your data ingestion, data landing, SQL schema creation, data transformation, and ODS and HDS creation/updates.
As an example of effective data structuring, Qlik works with a major provider of services to the automotive industry to more efficiently feed and transform data in a multi-zone data lake pipeline. The firm’s data is extracted from DB2 iSeries and then landed as raw deltas in an Amazon S3-based data lake. In the next S3 zone, tables are assembled (i.e., cleansed and merged) with a full persisted history available to identify potential errors and/or rewind, if necessary. Next these tables are provisioned/presented via point-in-time snapshots, ODS, and comprehensive change histories. Finally, analysts consume the data through an Amazon Redshift data warehouse. In this case, the data lake pipeline transformed the data while structured data warehouses perform the actual analysis. The firm is automating each step in the process.
A key takeaway here is that the most successful enterprises automate the deployment and continuous updates of multiple data zones to reduce time, labor, and costs. Consider the skill sets of your IT team, estimate the resources required, and develop a plan to either fully staff your project or use a technology that can reduce anticipated skill and resource requirements without compromising your ability to deliver.
Automating the Data Lake Pipeline
Your data management processes should minimize production impact and increase efficiency as your data volumes and supporting infrastructure grow. Quantities of hundreds or thousands of data sources affect implementation time, development resources, ingestion patterns (e.g., full data sets versus incremental updates), the IT environment, maintainability, operations, management, governance, and control.
Here again organizations find automation reduces time and staff requirements, enabling staff to efficiently manage ever- growing environments. Best practices include implementing an efficient ingestion process, eliminating the need for software agents on each source system, and centralizing management of sources, targets, and replication tasks across the enterprise.
With Qlik Replicate, your organization can scale to efficiently manage data flows across the world’s largest enterprise environments. Our zero-footprint architecture eliminates the need to install, manage, and update disruptive agents on sources or targets. In addition, Qlik Enterprise
Manager™ (formerly Attunity Enterprise Manager) is an intuitive and fully automated, single console to configure, execute, monitor, and optimize thousands of replication tasks across hundreds of end points. You can track key performance indicators (KPIs) in real time and over
time to troubleshoot issues, smooth performance, and plan the capacity of Qlik Replicate servers. The result: the highest levels of efficiency and scale.
Whenever possible, your organization should consider adopting specialized technologies to integrate data from mainframe, SAP, cloud, and other complex environments. Here’s why:
Enabling analytics with SAP-sourced data on external platforms requires decoding data from SAP pooled and clustered tables and enabling business use on a common data model. Cloud migrations require advanced performance and data encryption over WANs.
And deep integration with mainframe sources is needed to offload data and queries with sufficient performance. Data architects have to take these and other platform complexities into account when planning data lake integration projects.
Qlik Replicate provides comprehensive and deep integration with all traditional and legacy production systems, including Oracle, SAP, DB2 z/ OS, DB2 iSeries, IMS, and VSAM. Our company has invested decades of engineering to be able to easily and non-disruptively extract and decode transactional data, either in bulk or real time, for analytics on any major external platform.
When decision makers at an international food industry leader needed a current view and continuous integration of production-capacity data, customer orders, and purchase orders to efficiently process, distribute, and sell tens of millions of chickens each week, they turned to Qlik. The company had struggled to bring together its large datasets, which were distributed across several acquisition-related silos within SAP Enterprise Resource Planning (ERP) applications. The company relied on slow data extraction and decoding processes that were unable to match orders and production line-item data fast enough, snarling plant operational scheduling and preventing sales teams from filing accurate daily reports.
The global food company converted to a new Hadoop Data Lake based on the Hortonworks Data Platform and Qlik Replicate. It now uses our SAP-certified software to efficiently copy SAP record changes every five seconds, decoding that data from complex source SAP pool and cluster tables. Qlik Replicate injects this data stream – along with any changes to the source metadata and DDL changes – to a Kafka message queue that feeds HDFS and HBase consumers subscribing to the relevant message topics (one topic per source table).
Once the data arrives in HDFS and HBase, Spark in-memory processing helps match orders to production on a real-time basis and maintain referential integrity for purchase order tables within HBase and Hive. The company has accelerated sales and product delivery with accurate real-time operational reporting. Now, it operates more efficiently and more profitably because it unlocked data from complex SAP source structures.
The global food company converted to a new Hadoop Data Lake based on the Hortonworks Data Platform and Qlik Replicate. It now uses our SAP-certified software to efficiently copy SAP record changes every five seconds, decoding that data from complex source SAP pool and cluster tables. Qlik Replicate injects this data stream – along with any changes to the source metadata and DDL changes – to a Kafka message queue that feeds HDFS and HBase consumers subscribing to the relevant message topics (one topic per source table).
Once the data arrives in HDFS and HBase, Spark in-memory processing helps match orders to production on a real-time basis and maintain referential integrity for purchase order tables within HBase and Hive. The company has accelerated sales and product delivery with accurate real-time operational reporting. Now, it operates more efficiently and more profitably because it unlocked data from complex SAP source structures.
Streaming Data to a Cloud-based Data Lake and Data Warehouse
By adhering to these five principles, your enterprise IT organization can strategically build an architecture on premises or in the cloud to meet historical and real-time analytics requirements. Our solution, which includes Qlik Replicate and Qlik Compose for Data Lakes, addresses key challenges and moves you closer to achieving your business objectives.
The featured case studies and this sample architecture and description show how Qlik manages data flows at each stage of a data lake pipeline.
Your Data Lake Pipeline
Take a closer look, starting with the Landing Zone. First, Qlik Replicate copies data – often from traditional sources such as Oracle, SAP, and mainframe – then lands it in raw form in the Hadoop File System. This process illuminates all the advantages of Qlik Replicate, including full load/CDC capabilities, time-based partitioning for transactional consistency and auto-propagation of source DDL changes. Now, data is
ingested and available as full snapshots or change tables, but not yet ready for analytics.
In the Assemble Zone, Qlik Compose for Data Lakes standardizes and combines change streams into a single transformation-ready data store. It automatically merges the multi-table and/or multi-sourced data into a flexible format and structure, retaining full history to rewind and identify/remediate bugs, if needed. The resulting persisted history provides consumers with rapid access to trusted data, without having to understand or execute the structuring that has taken place. Meanwhile, you, your data managers, and architects maintain central control of the entire process.
In the Provision Zone, your data managers and architects provision an enriched data subset to a target, potentially a structured data warehouse, for consumption (curation, preparation, visualization, modeling, and analytics) by your data scientists and analysts. Data can be continuously updated to these targets to maintain fresh data.
Our Qlik software also provides automated metadata management capabilities to help your enterprise users better understand, utilize, and trust their data as it flows into and is transformed within their data lake pipeline. With our Qlik Replicate and Qlik Compose solutions you can add, view, and edit entities (e.g., tables) and attributes (i.e., columns). Qlik Enterprise Manager centralizes all this technical metadata so anyone can track the lineage of any piece of data from source to target, and assess the potential impact of table/column changes across data zones. In addition, Qlik Enterprise Manager collects and shares operational metadata from Qlik Replicate with third-party reporting tools for enterprise-wide discovery and reporting. And our company continues to enrich our metadata management capabilities and contribute to open-source industry initiatives such as ODPi to help simplify and standardize Big Data ecosystems with common reference specifications.
You improve the odds of data lake success by planning and designing for platform flexibility, data in motion, automation, scalability, and deep source integration. Most important, each of these principles hinge on effective data integration capabilities.
Our Qlik technology portfolio accelerates and automates data flows across your data lake pipeline, reducing your time to analytics readiness. It provides efficient and automated management of data flows and metadata. Using our software, you and your organization can improve SLAs, eliminate data and resource bottlenecks, and more efficiently manage higher-scale data lake initiatives. Get your analytics project back on track and help your business realize more value faster from your data with Qlik.
Next Steps
For more information or enquiries about Qlik products and services, feel free to contact us below.
Connect with SIFT Analytics
As organisations strive to meet the demands of the digital era, SIFT remains steadfast in its commitment to delivering transformative solutions. To explore digital transformation possibilities or learn more about SIFT’s pioneering work, contact the team for a complimentary consultation. Visit the website at www.sift-ag.com for additional information.
About SIFT Analytics
Get a glimpse into the future of business with SIFT Analytics, where smarter data analytics driven by smarter software solution is key. With our end-to-end solution framework backed by active intelligence, we strive towards providing clear, immediate and actionable insights for your organisation.
Headquartered in Singapore since 1999, with over 500 corporate clients, in the region, SIFT Analytics is your trusted partner in delivering reliable enterprise solutions, paired with best-of-breed technology throughout your business analytics journey. Together with our experienced teams, we will journey. Together with you to integrate and govern your data, to predict future outcomes and optimise decisions, and to achieve the next generation of efficiency and innovation.
The Analytics Times
“The Analytics Times is your source for the latest trends, insights, and breaking news in the world of data analytics. Stay informed with in-depth analysis, expert opinions, and the most up-to-date information shaping the future of analytics.
Published by SIFT Analytics
SIFT Marketing Team
marketing@sift-ag.com
+65 6295 0112
SIFT Analytics Group
Explore our latest insights
Data is an extremely valuable asset to almost every organization, and it informs nearly every decision an enterprise makes. It can be used to make better decisions at almost every level of the enterprise—and to make them more quickly. But to take full advantage of the data and to
do so quickly requires artificial intelligence (AI). So, it is no surprise that nearly all participants in our research (87%) report that they have enabled or piloted AI features in analytics and business intelligence applications. Today, data is collected in more ways and from more
devices and more frequently than ever before. It can enable new methods of doing business and can even create new sources of revenue. In fact, the data and analyses themselves can be a new source of revenue.
Independent software vendors (ISVs) and data providers understand the importance of data in AI-based processes, and they are designing products and services to help enterprises step in and harness all this data and AI-generated business energy. To maximize the opportunities,
ISVs and data providers need to recognize that enterprises use various types of data, including data from both internal and external
sources. In fact, our research shows that the majority of enterprises (56%) are working with 11 or more sources of data. Governing the various data sources becomes critical because poor quality data leads to poor AI models. Our research shows the top benefit of investing in data governance, reported by three-quarters of participants (77%), is improved data quality.
The most common types of collected data include transactional, financial, customer, IT systems, employee, call center, and supply
chain. But there are other sources as well, many external to the enterprise. Nine in 10 enterprises (90%) are working with at least one
source of external data, which could mean location data, economic data, social media, market data, consumer demographics government data, and weather data. To be useful, all of that must be integrated.
“Data integration” is the process of bringing together information from various sources across an enterprise to provide a complete, accurate, and real-time set of data that can support
operational processes and decision-making. But nearly one-third of enterprises (31%) report that it is hard to access their data sources, and more than two-thirds (69%) report that preparing their data is the activity where they spend the most time in their analytics
processes. The process of data integration often places a burden on the operational systems upon which enterprises rely.
At the same time, enterprises also need to be able to integrate applications into their data processes. ISVs and data providers must bring data together with applications so it is easier for enterprises to access and use the very data they provide.
Simple linkages such as open database connectivity and Java database connectivity (ODBC/JDBC), or even custom-coded scripts, are not sufficient for data integration. While ODBC/JDBC can provide the necessary “plumbing” to access many different data sources, it offers little assistance to application developers in creating agile data pipelines. Simple connectivity also does nothing to assist with consolidating or transforming data to make it ready for analytics, for instance, in a star schema. Nor does simple connectivity provide any assistance in dealing with slowly changing dimensions which must be tracked for many types of AI analyses.
Simple connectivity does little to help enterprises transform the data to ensure its standardization or quality. Data from various sources often contains inconsistencies, for instance in customer reference numbers or product codes. Accurate analyses require that these inconsistencies be resolved as the data is integrated. Similarly, data quality is an issue that must be addressed as the data is integrated. Our research shows these two issues of data quality and consistency are the second most common time sinks in the analytics process.
Nor does simple database connectivity help enterprises effectively integrate data from files, applications or application programming interfaces (APIs). With the proliferation of cloudbased applications, many of which only provide API access, ODBC/JDBC connectivity may not be an option. And many enterprises still need to process flat files of data, as our research shows that these types of files are the second most common source of data for analytics.
Data integration is not a one-time activity, either. It requires the establishment of data pipelines that regularly collect and consolidate
updated data. A greater infrastructure is needed around these pipelines to ensure that they run properly and to completion. ISVs and data providers that rely only on simple connectors must create and maintain this extra infrastructure themselves.
Those data pipelines also need to be agile enough to support a variety of styles of integration. Batch updates are still useful for bulk transfers of data, but other more frequent styles of updating are needed as well. Our research shows that nearly one-quarter of enterprises (22%) need to analyze data in real time. Since the most common sources of information are transactional and operational applications, it is important to create pipelines that can access this data as it is generated. Incremental updates and change data capture (CDC) technology can solve this problem and these are becoming competitive necessities.
Real-time requirements are even more demanding when we consider event data, where nearly one-half (47%) of enterprises process it within seconds. Then, as applications and organizational requirements change, the data pipelines must reflect those changes. Therefore, the tools used to support such a wide variety of ever-changing sources need to be open enough to be easily incorporated into a wide variety of processes.
But if ISVs and data providers focus their energies on maintaining data pipelines, it distracts resources from the core business. Creating data pipeline infrastructure that is highly performant and efficient requires years of engineering. Simple bulk movement of entire data sets is slow and inefficient, even though it may be necessary for initial data transfers. Subsequent data transfers, however, should use a data replication scheme or CDC approach, creating much smaller data transfers and much faster processes.
A modern data fabric is based on a cloud-native architecture and includes orchestration and automation capabilities that enhance the design and execution of data pipelines that consolidate information from across the enterprise. As data becomes a new source of revenue, sometimes referred to as “data as a product,” a modern data fabric must also enable easy access to, and consumption of, data. A key component to delivering data in this fashion is strong data catalog capabilities. AI assisted search, automated profiling and tagging of data sources, and tracking the lineage of that data through its entire life cycle make it easier to find and understand the data needed for particular operations and analyses. Collecting and sharing this metadata in a data catalog not only provides better understanding and access to the data, but also improves data governance. Our research shows that enterprises that have adequate data catalog technology are three times more likely to be satisfied with their analytics and have achieved greater rates of self-service analytics.
Orchestration and access via APIs are also critical to ISVs and data providers as these allow the remote invocation of data pipelines needed for the coordination and synchronization of various interrelated application processes, even when they are distributed across different cloud applications and services. These APIs need to span all aspects from provisioning to core functionality for orchestration to be effective. Automation of these orchestration tasks can enhance many aspects of data pipelines to make them both more efficient and more agile.
Automated data mapping, automated meta data creation and management, schema evolution, automated data mart creation, and data warehouse and data lake automation can quickly and efficiently create analytics-ready data. When combined with orchestration, automation can also provide “reverse integration” to update data in source systems when necessary and appropriate.
Modern data integration platforms employ AI/ML to streamline and improve data processing. AI/ML can be used to automatically detect anomalies in data pipelines, such as whether the pipelines suddenly processed an unusually small number of records. Such an anomaly could indicate a problem somewhere else in the pipeline. AI/ML can also be used to automatically deal with errors in pipelines and routine changes, such as those in the sources or targets. AI/ML can also determine the optimal execution of pipelines, including the number of instances to create or where different portions of the pipeline should be processed. AI/ML can be used to enrich data with predictions, scoring or classifications that help support more accurate decision-making. We assert that by 2027, three-quarters of all data processes will use AI and ML to accelerate the realization
of value from the data.
Modern data integration platforms must also incorporate all appropriate capabilities for data governance. Data sovereignty issues may require that data pipelines be executed only within certain geographies. Compliance with internal or regulatory policies may require single sign-on or the use of additional credentials to appropriately track and govern data access and use. Therefore, a platform with built-in capabilities for governance can help identify personally identifiable information and other sensitive or regulated data. But implementing any of these modern data integration platform requirements can impose a significant burden on ISVs and data providers.
Product Distributors
For organizations with hundreds of thousands of SKUs and hundreds of thousands of customers, managing orders and inventories can be a time consuming process. Using a modern data-as-a-product approach with standardized data governance and a centralized data catalog can reduce costs dramatically and enable self-service online ordering. This approach also creates more agility to meet customer needs and provides better, more timely visibility into operations.
Insurance Industry
Insurance technology data providers can use data integration to help their customers be more competitive by providing access to up-to-date information that enables online quotes. Data is the key to the accurate pricing of insurance liabilities, and many of the sources and targets exist in the cloud, but they require support for a variety of endpoints. By using CDC-based replication, however, both claims and market data can be collected, consolidated, and distributed within minutes. As a result, millions of quotes can be generated each day where each incorporates real-time analysis of vast volumes of data.
Other Applications
Data integration can be the key to many other ISVs and data providers. Mobile application providers can integrate location data with transaction data to provide broader market data on consumer behavior. Talent management ISVs can integrate data relating to internal performance and compensation with external market data to improve employee acquisition and retention. Foreclosure data can be collected, consolidated, and distributed to support loan origination and servicing operations. Vendor data can be collected and provided to improve procurement processes augmenting supplier performance analyses with risk, diversity, sustainability and credit scores. And regardless of the vertical industry or line-of-business function, faster access to more data generally produces better results.
Once data is integrated, it can provide the basis for a broad range of analytics and AI. By supporting these analyses and data science, ISVs and data providers can extend the value of their capabilities and therefore increase their revenue opportunities. Choosing a data integration platform that also supports analytics and AI will make it easier for enterprises to capture this revenue. In fact, our research shows that reports and dashboards are the most common types of analytics used by more than 80% of enterprises. However, when considering analytics providers, look at those that support other newer techniques as well, such as AI/ML and natural language processing, which are projected to be required by 80% of enterprises in the future.
Enterprises need to use data to help drive actions. Data can help them understand what has happened and why, but they ultimately need to process what they have learned and then take action. In many situations, however, there is simply no time to review data to determine what
course of action to take. ISVs and data providers can help their customers derive more value from data by using real-time information to trigger the appropriate actions.
ISVs and data providers are using technology to add value to business processes. While all business processes typically require data, data integration itself is merely a means to the end. If the process is not done properly, it can detract from the overall approach, so it requires careful design and development. Enterprises should ideally spend their time on core competencies, not on developing data integration technology. By using a full-featured, purpose-built data integration platform, they can ensure that the data needed by ISVs and data providers is robust and available in a timely manner.
Connect with SIFT Analytics
As organisations strive to meet the demands of the digital era, SIFT remains steadfast in its commitment to delivering transformative solutions. To explore digital transformation possibilities or learn more about SIFT’s pioneering work, contact the team for a complimentary consultation. Visit the website at www.sift-ag.com for additional information.
About SIFT Analytics
Get a glimpse into the future of business with SIFT Analytics, where smarter data analytics driven by smarter software solution is key. With our end-to-end solution framework backed by active intelligence, we strive towards providing clear, immediate and actionable insights for your organisation.
Headquartered in Singapore since 1999, with over 500 corporate clients, in the region, SIFT Analytics is your trusted partner in delivering reliable enterprise solutions, paired with best-of-breed technology throughout your business analytics journey. Together with our experienced teams, we will journey. Together with you to integrate and govern your data, to predict future outcomes and optimise decisions, and to achieve the next generation of efficiency and innovation.
The Analytics Times
“The Analytics Times is your source for the latest trends, insights, and breaking news in the world of data analytics. Stay informed with in-depth analysis, expert opinions, and the most up-to-date information shaping the future of analytics.
Published by SIFT Analytics
SIFT Marketing Team
marketing@sift-ag.com
+65 6295 0112
SIFT Analytics Group
Explore our latest insights
ในยุคที่ข้อมูล (Data) เข้ามามีบทบาทสำคัญในการพัฒนาองค์กร ส่งผลให้การเรียนรู้ของเครื่อง (Machine Learning) และ วิทยาศาสตร์ข้อมูล (Data Science) เข้ามามีผลในการวิเคราะห์ข้อมูล (Data Analytics) ธุรกิจขององค์กรและบริษัทเพิ่มมากขึ้นเรื่อยๆ แต่อย่างไรก็ตามไม่ใช่ทุกองค์กรหรือบริษัทที่จะมีทีมนักวิทยาศาสตร์ข้อมูล (Data Scientist) ที่จะทำงานทางด้านนี้โดยตรง หรือบางองค์กรก็มีทีมขนาดเล็ก ส่งผลให้เมื่อองค์กรต้องการพัฒนาโครงการวิทยาศาสตร์ข้อมูลหรือ Machine Learning 100 โครงการ ทีมนักวิทยาศาสตร์ข้อมูลจะสามารถมุ่งเน้นได้แค่ 10 อันดับแรก ที่ส่งผลกระทบใหญ่กับธุรกิจหลักโดยตรง และมีกระบวนการขั้นตอนวิธีการที่ลึกซึ้ง ซับซ้อนมาก ใช้ทรัพยากรและเวลามาก แล้วโครงการอื่นๆ ที่เหลือละ? จะปล่อยมันทิ้งไว้เฉย ๆ เลยหรือ ?
จึงเป็นที่มาของ Qlik AutoML ที่เข้ามาส่งเสริมช่วยเหลือรองรับโครงการอีก 90% ที่เหลือ เพื่อช่วยให้นักวิเคราะห์ข้อมูล (Data Analyst) และนักวิเคราะห์ธุรกิจ (Business Analyst) ของคุณสามารถวิเคราะห์ข้อมูลเชิงลึกเชิงคาดการณ์ (Predictive Insight) เพื่อนำไปแก้ไขปัญหาทางธุรกิจได้เพิ่มมากขึ้น
Machine Learning (ML) เป็นสาขาหนึ่งของนวัตกรรมปัญญาประดิษฐ์ (AI) ที่มุ่งเน้นไปที่กระบวนการรับรู้รูปแบบในข้อมูลในอดีตเพื่อคาดการณ์ผลลัพธ์ในอนาคต โดยใช้ข้อมูลที่สังเกตได้ในอดีตเป็นอินพุต และใช้กระบวนการทางคณิตศาสตร์กับข้อมูลเหล่านั้น เพื่อสร้างเอาต์พุต ที่เรียกว่าโมเดลการเรียนรู้ของระบบคอมพิวเตอร์ ตามรูปแบบข้อมูลในอดีต ซึ่งโมเดลนี้สามารถใช้เพื่อคาดการณ์และทดสอบสถานการณ์ในอนาคตได้
ส่วน AutoML (Auto Machine Learning) หรือ Machine Learning แบบอัตโนมัติ เป็นแนวทาง Machine Learning ที่ทำให้ไปป์ไลน์ข้อมูล ขั้นตอนการประมวลผลข้อมูล การเลือกคุณสมบัติ การเลือกอัลกอริทึม การฝึกโมเดล การโฮสต์โมเดล และกระบวนการปรับใช้ผ่านการใช้ AI เป็นแบบอัตโนมัติ ซึ่งถูกออกแบบมาโดยเฉพาะสำหรับทีมวิเคราะห์ข้อมูล
และ Qlik AutoML จะช่วยให้นักวิเคราะห์และทีมวิเคราะห์สามารถคาดการณ์เกี่ยวกับผลลัพธ์ทางธุรกิจ ทำความเข้าใจว่าทำไมการคาดการณ์เหล่านั้นจึงเกิดขึ้น และสร้างการดำเนินการที่มีประสิทธิภาพสูงสุดเพื่อกระตุ้นให้เกิดผลลัพธ์เหล่านั้น หรือที่เราเรียกกันว่า Prescriptive Analytics
Qlik AutoML โซลูชันสำหรับการวิเคราะห์ข้อมูลที่เหมาะสมกับทุกบทบาททางธุรกิจ พร้อมพัฒนาและปรับแต่งได้ตามความต้องการขององค์กร
ที่มา: Qlik AutoML Machine Learning for Analytics Teams 2023,10
การขาย (Sales)
การตลาด (Marketing)
การเงิน (Finance)
ฝ่ายปฏิบัติการ (Operations)
ฝ่ายบุคคล (HR)
ฝ่ายเทคโนโลยีสารสนเทศ (IT)
ฝ่ายผลิตและห่วงโซอุปทาน (Supply Chain)
ฝ่ายบริการและการสนับสนุน (Service & Support)
ที่มา: Qlik AutoML Machine Learning for Analytics Teams 2023,18
Qlik Cloud นำเสนอการทำงานของ AutoML ตั้งแต่ต้นจนจบ โดยสามารถสร้างและนำโมเดลไปใช้งานโดยง่าย ไม่ต้องเขียนโค้ด
Qlik AutoML รองรับการทำงานตั้งแต่ขั้นตอนจัดเตรียมข้อมูล เลือกข้อมูลที่สำคัญ สร้างโมเดล ฝึก ปรับแต่ง นำโมเดลไปใช้ในการพยากรณ์ข้อมูลพร้อมอธิบาย จากนั้นบูรณาการเชื่อมต่อกับ Qlik Sense แอพและแดชบอร์ด เพื่อวิเคราะห์ข้อมูลเชิงตอบโต้ และวิเคราะห์ what-if analysis ทำให้ลูกค้าสามารถวางแผนการตัดสินใจและการดำเนินการทางธุรกิจ
ด้วยความสามารถอัจฉริยะของ Qlik AutoML จึงทำให้มันแตกต่างจาก Machine Learning ที่คุณคุ้นเคย ไม่ว่าจะเป็น
ซึ่งความสามารถเหล่านี้จะทำให้การทำงานของคุณง่ายขึ้น รวดเร็วขึ้น และสามารถขับเคลื่อนองค์กรด้วยการสร้างการตัดสินใจอย่างแม่นยำมากยิ่งขึ้นนั่นเอง หากสนใจเรียนรู้เพิ่มเติมเกี่ยวกับ Qlik AutoML สามารถติดต่อทีมงานของ Sift Analytics Group ได้เลย
เพราะ ที่ Sift Analytics Group เราไม่เพียงแค่จัดจำหน่ายซอฟท์แวร์สำหรับการวิเคราะห์ข้อมูล แต่เรายังมีบริการครอบคลุมถึงขั้นตอนการให้คำแนะนำเพื่อนำธุรกิจของท่านไปสู่ความสำเร็จจากการวิเคราะห์ข้อมูลที่แม่นยำและสร้างการตัดสินใจที่ถูกต้องอีกด้วย
ที่มาของเนื้อหา:
Qlik AutoML Machine Learning for Analytics Teams 2023
https://www.qlik.com/us/-/media/files/resource-library/global-us/direct/datasheets/ds-qlik-automl-machine-learning-for-your-analytics-teams-en.pdf?rev=16d86fa7318e4a32b4290a65e7d296fc
เชื่อมต่อกับ SIFT Analytics
ในยุคที่องค์กรต่าง ๆ ต้องการก้าวให้ทันทุกความเปลี่ยนแปลงด้านดิจิทัล SIFT เองยังคงมุ่งมั่นในการนำเสนอโซลูชันที่จะมาสร้างการเปลี่ยนแปลงทางธุรกิจให้กับลูกค้าของเราอยู่เสมอ หากต้องการเรียนรู้เพิ่มเติมเพื่อเริ่มต้นการเปลี่ยนแปลงทางดิจิทัลกับ SIFT สามารถติดต่อทีมงานของเราเพื่อขอคำปรึกษาฟรี หรือยี่ยมชมเว็บไซต์ของเราเพื่อค้นหาข้อมูลเพิ่มเติมได้ที่ www.sift-ag.com
เกี่ยวกับ SIFT Analytics
ก้าวทันอนาคตของธุรกิจอย่างชาญฉลาดยิ่งขึ้นด้วย SIFT Analytics ผู้ให้บริการการวิเคราะห์ข้อมูลซึ่งขับเคลื่อนโดยโซลูชันซอฟต์แวร์แบบครบวงจรและถูกพัฒนาด้วยระบบอัฉริยะเชิงรุก (Active intelligence) ที่ล้ำสมัย เรามุ่งมั่นที่จะมอบข้อมูลเชิงลึกที่ชัดเจนและนำไปใช้ได้ทันทีสำหรับองค์กรของคุณ
เรามีสำนักงานใหญ่ในสิงคโปร์ตั้งแต่ปี 1999 และมีลูกค้าองค์กรมากกว่า 500 รายในภูมิภาค SIFT Analytics เป็นพันธมิตรที่เชื่อถือได้ของคุณในการนำเสนอโซลูชันระดับองค์กรที่เชื่อถือได้ จับคู่กับเทคโนโลยีที่ดีที่สุดตลอดเส้นทางการวิเคราะห์ธุรกิจของคุณ เราจะเดินทางร่วมกับทีมงานที่มีประสบการณ์ของเรา ร่วมกับคุณเพื่อบูรณาการและควบคุมข้อมูลของคุณ เพื่อคาดการณ์ผลลัพธ์ในอนาคตและเพิ่มประสิทธิภาพการตัดสินใจ และเพื่อให้บรรลุประสิทธิภาพและนวัตกรรมรุ่นต่อไป
เผยแพร่โดย SIFT Analytics – SIFT Thailand Marketing Team
Trong thời đại số hóa, quyết định dựa trên dữ liệu chính xác và trực quan là chìa khóa để đưa doanh nghiệp đến thành công. Qlik Sense, một giải pháp trực quan hoá dữ liệu hàng đầu. Không chỉ là công cụ, mà là một trải nghiệm đưa bạn đến với sức mạnh của thông tin.
1. Trực quan hoá dữ liệu nhanh chóng
Qlik Sense không chỉ giúp bạn hiểu dữ liệu một cách nhanh chóng mà còn biến chúng thành các biểu đồ, đồ thị phức tạp một cách mạch lạc.
2. Khả năng kết nối đa nguồn dữ liệu
Với khả năng kết nối đa nguồn dữ liệu, từ bảng Excel đến cơ sở dữ liệu lớn, Qlik Sense giúp tạo nên cái nhìn toàn diện về dữ liệu.
3. Tích hợp học máy
Qlik Sense không chỉ dừng lại ở việc hiển thị dữ liệu mà còn tích hợp công nghệ học máy, giúp dự đoán xu hướng và phân tích sâu sắc.
4. Trải nghiệm người dùng mạnh mẽ
Giao diện trực quan và thân thiện với người dùng, giúp mọi thành viên trong doanh nghiệp có thể sử dụng Qlik Sense một cách dễ dàng.
5. Bảo mật cao
Với các tính năng bảo mật tiên tiến, Qlik Sense đảm bảo rằng dữ liệu của bạn được bảo vệ tối đa mà vẫn đảm bảo tính linh hoạt.
Dữ liệu trực quan hoá All – in – one
Tất cả dữ liệu được chuyển hoá thành hình ảnh và được số hoá chi tiết.
Hiểu Rõ Hơn
Dữ liệu trực quan giúp mọi người hiểu rõ hơn về bức tranh tổng thể và chi tiết.
Ra Quyết Định Nhanh Chóng
Có thể đưa ra quyết định dựa trên dữ liệu một cách nhanh chóng và chính xác.
Tạo Đà Cho Sự Đổi Mới
Trực quan hoá dữ liệu là bước đầu tiên để tạo đà cho sự đổi mới và sáng tạo trong doanh nghiệp.
Tăng Hiệu Quả Công Việc
Giúp tăng cường hiệu quả công việc và giảm thời gian xử lý thông tin.
Qlik Sense sẽ không chỉ là lựa chọn, mà là sự đầu tư đúng đắn cho sự phát triển của doanh nghiệp. Đặt bước đi chính xác với Qlik Sense hôm nay!
Khám phá những thông điệp mới nhất của chúng tôi
Philadelphia – Recent research by Qlik shows that enterprises are planning significant investments in technologies that enhance data fabrics to enable generative AI success, and are looking to a hybrid approach that incorporates generative AI with traditional AI to scale its impact across their organizations.
The “Generative AI Benchmark Report”, executed in August 2023 by Enterprise Technology Research (ETR) on behalf of Qlik, surveyed 200 C-Level executives, VPs, and Directors from Global 2000 firms across multiple industries. The survey explores how leaders are leveraging the generative AI tools they purchased, lessons learned, and where they are focusing to maximize their generative AI investments.
“Generative AI’s potential has ignited a wave of investment and interest both in discreet generative AI tools, and in technologies that help organizations manage risk, embrace complexity and scale generative AI and traditional AI for impact,” said James Fisher, Chief Strategy Officer at Qlik. “Our Generative AI Benchmark report clearly shows leading organizations understand that these tools must be supported by a trusted data foundation. That data foundation fuels the insights and advanced use cases where the power of generative AI and traditional AI together come to life.”
The report found that while the initial excitement of what generative AI can deliver remains, leaders understand they need to surround these tools with the right data strategies and technologies to fully realize their transformative potential. And while many are forging ahead with generative AI to alleviate competitive pressures and gain efficiencies, they are also looking for guidance on where to start and how to move forward quickly while keeping an eye on risk and governance issues.
Creating Value from Generative AI
Even with the market focus on generative AI, respondents noted they clearly see traditional AI still bringing ongoing value in areas like predictive analytics. Where they expect generative AI to help is in extending the power of AI beyond data scientists or engineers, opening up AI capabilities to a larger population. Leaders expect this approach will help them scale the ability to unlock deeper insights and find new, creative ways to solve problems much faster.
This vision of what’s possible with generative AI has driven an incredible level of investment. 79% of respondents have either purchased generative AI tools or invested in generative AI projects, and 31% say they plan to spend over $10 million on generative AI initiatives in the coming year. However, those investments run the risk of being siloed, since 44% of these organizations noted they lack a clear generative AI strategy.
Surrounding Generative AI with the Right Strategy and Support
When asked how they intend to approach generative AI, 68% said they plan to leverage public or open-source models refined with proprietary data, and 45% are considering building models from scratch with proprietary data.
Expertise in these areas is crucial to avoiding the widely reported data security, governance, bias and hallucination issues that can occur with generative AI. Respondents understand they need help, with 60% stating they plan to rely partially or fully on third-party expertise to close this gap.
Many organizations are also looking to data fabrics as a core part of their strategy to mitigate these issues. Respondents acknowledged their data fabrics either need upgrades or aren’t ready when it comes to generative AI. In fact, only 20% believe their data fabric is very/extremely well equipped to meet their needs for generative AI.
Given this, it’s no surprise that 73% expect to increase spend on technologies that support data fabrics. Part of that spend will need to focus on managing data volumes, since almost three quarters of respondents said they expect generative AI to increase the amount of data moved or managed on current analytics. The majority of respondents also noted that data quality, ML/AI tools, data governance, data integration and BI/Analytics all are important or very important areas to delivering a data fabric that enables generative AI success. Investments in these areas will help organizations remove some of the most common barriers to implementation per respondents, including regulation, data security and resources.
The Path to Generative AI Success – It’s All About the Data
While every organization’s AI strategy can and should be different, one fact remains the same: the best AI outcomes start with the best data. With the massive amount of data that needs to be curated, quality-assured, secured, and governed to support AI and construct useful generative AI models, a modern data fabric is essential. And once data is in place, the platform should deliver end-to-end, AI-enabled capabilities that help all users – regardless of skill level – get powerful insights with automation and assistance. Qlik enables customers to leverage AI in three critical ways:
To learn more about how Qlik is helping organizations drive success with AI, visit Qlik Staige.
Connect with SIFT Analytics
As organisations strive to meet the demands of the digital era, SIFT remains steadfast in its commitment to delivering transformative solutions. To explore digital transformation possibilities or learn more about SIFT’s pioneering work, contact the team for a complimentary consultation. Visit the website at www.sift-ag.com for additional information.
About SIFT Analytics
Get a glimpse into the future of business with SIFT Analytics, where smarter data analytics driven by smarter software solution is key. With our end-to-end solution framework backed by active intelligence, we strive towards providing clear, immediate and actionable insights for your organisation.
Headquartered in Singapore since 1999, with over 500 corporate clients, in the region, SIFT Analytics is your trusted partner in delivering reliable enterprise solutions, paired with best-of-breed technology throughout your business analytics journey. Together with our experienced teams, we will journey. Together with you to integrate and govern your data, to predict future outcomes and optimise decisions, and to achieve the next generation of efficiency and innovation.
Published by SIFT Analytics
SIFT Marketing Team
marketing@sift-ag.com
+65 6295 0112
SIFT Analytics Group
Explore our latest insights
