When SAP Visual Intelligence was introduced in 2012, it could visualize data from any data source as long as it was SAP HANA. Thankfully, universe support came just a few short months later in version 1.03, opening up a wealth of data sources and allowing customers to leverage their existing investments in SAP’s “agnostic” semantic layer. Universe support in Lumira is interesting because the front-end doesn’t give the full BusinessObjects query panel experience that experienced universe consumers expect. But it’s also interesting because of how data gets from the data source, through the universe, and into Lumira. It uses a Web Intelligence mechanism originally designed to allow Web Intelligence documents to be exported from the browser in Microsoft Excel and text formats.
The SAP Lumira universe query panel
First, let’s take a look at the query panel experience. First, here’s the Web Intelligence Java-based universe query panel (from BI 4.1 SP3).
And here’s the Lumira universe query panel (from version 1.19).
At first glance I assumed that Lumira was offering some kind of reimagined query panel of the future. But upon closer inspection it’s a query panel that’s missing quite a few features from Web Intelligence panel, similar to Explorer (see related article, Family Planning), Live Office, and Query as a Web Service (see related article, What I miss in the Query as a Web Service (QaaWS) and Live Office query panels). Even Design Studio has more query panel functionality (although still a subset of the gold standard Web Intelligence panel). It’s possible that the Lumira product team assumed that its users would want something easier to use. But consider that SAP Lumira already has a free-hand SQL capability that’s still lacking from Web Intelligence. Free-hand SQL provides many things, but “easy business user interface” doesn’t immediately spring to mind (see related article, Free-Hand SQL Isn’t Free).
There definitely is an opportunity here to extend the functionality of the Lumira universe query panel.
UPDATE: Lumira v1.24 will integrate the Information Design Tool’s Query Panel, but only for UNX universes.
Consuming Existing Universe Queries
I’ve already made the unsuccessful pitch that Lumira should be able to extract queries from Desktop Intelligence documents (see related article, True Desktop Intelligence with SAP Lumira). And the longer I work with Lumira, it seems obvious that it should be able to consume a query from a Web Intelligence document, too. There are obvious differences in functionality between Web Intelligence and Lumira. Reporting is not data discovery and I’m not proposing to change that. But whether seconds, minutes, hours, or days were required to create a critical Web Intelligence report, it seems logical that a Lumira user might want to consume the same query logic without reinventing the wheel.
UPDATE (June 9, 2015): This idea was intriguing enough to APOS, who has developed the APOS Data Gateway plug-in for SAP Lumira.
SAP Lumira and the Web Intelligence processing server character stream size
The other issue Lumira users will encounter while squeezing the entire universe into their visualizations is the Web Intelligence Processing Server maximum character stream size. Experienced SAP BI administrators refer to this as the “10 kilograms of universe, uh, DATA in a 5 kilogram bag” problem. Users will see the following message and probably have their own nickname for it.
The following verbiage first appeared in the SAP Lumira 1.17 Release Notes. It was removed from the SAP Lumira 1.18 Release Notes but has been fortunately documented by SAP Note 2020352.
The data acquisition of medium-large, large, and very large datasets from UNV or UNX universes is not supported on default installations of both SAP BusinessObjects Business Intelligence platforms 4.0 and 4.1.
It is recommended for customers that want to acquire such datasets to install a SAP BusinessObjects Business Intelligence platform (SAP BIP) server dedicated to SAP Lumira and significantly increase the value of the Maximum Character Stream Size of the Web Intelligence Processing Server on that particular server. Note that increasing this value on a running SAP BIP server can impact the memory consumption and performance of any BI clients tool running on that installation, particularly Web Intelligence. While this practice is not formally discouraged, we advise customers implementing this solution that they might face memory consumption increases and longer document data refreshes so they will need to monitor the SAP BIP server’s behavior adequately to control the impacts.
In an era of “big data” hype, I have no idea what SAP means by the terms medium-large, large, and very large datasets, other than “don’t blame us if you didn’t buy a HANA server”. But as with Mobile BI (see related article, Viewing Large Web Intelligence Documents with Mobile BI), Explorer (see related article, Hacking SAP BusinessObjects Explorer 4.0) and Live Office, the solution is to increase the Maximum Character Stream Size and to a lesser extent the Binary Stream Maximum Size on the Web Intelligence Processing Server. SAP’s documentation is slightly unclear on this point, but it seems that XML and Microsoft Excel XLSX formats (which are zipped XML files- see related Wikipedia article about Office Open XML) are “character” files affected by the Maximum Character Stream Size setting. Adobe PDF and Microsoft Excel XLS formats are “binary” files affected by the adjacent Binary Stream Maximum Size setting. There isn’t an easy way to determine the optimal stream size, other than “keep increasing the value until the error goes away”. Keep in mind that poorly designed universes will return bloated data sets to any client tool, whether it’s SAP Lumira, Mobile BI, or any edition of Web Intelligence. So a code review of existing universes can be a healthy activity in addition to increasing server settings.
Oddly enough, Web Intelligence and its predecessor Desktop Intelligence contain SAP’s first in-memory database- the microcube. Long before HANA was a gleam in Hasso Plattner’s eye, the microcube facilitated multi-dimensional analysis of large datasets (called slicing-and-dicing back in the day) that may have taken a bit of time to be retrieved from the now-obsolete spinning disks in the database server. The Web Intelligence web application requests data from the microcube one viewable page at a time, but modern apps like Mobile BI and Lumira need the entire microcube before they can visualize data. So these apps are riding the coattails of the mechanism SAP originally created for exporting Web Intelligence data to Adobe PDF and Microsoft Excel files.
In a recent blog, Steve Lucas said
Build on what you already have: With 45,000 customers using SAP BI technology, there’s a pretty good chance you can take advantage of your existing BI investments by connecting directly to Universes and SAP Business Warehouse with SAP Lumira.
Steve Lucas, The Countdown on Data Visualization Has Begun – Top Ten Reasons Why SAP Will Deliver
If building on what we already have is a key selling point, there’s got to be a better, faster and more scalable way to get data from the universe into SAP Lumira.
I eagerly look forward to what is coming next from the SAP Lumira product team.
Is your organization using SAP Lumira with universes?
How these settings affect SAP Lumira
- SAP KB 1826015– SAP Visual Intelligence 1.0.7 connecting to universe data source
- SAP Note 2020352– Release restriction note for SAP Lumira 1.17
How these settings affect other SAP BusinessObjects products
- SAP KB 1527168– ERR_WIS_30272 when using LiveOffice towards a Universe Query
- SAP KB 1700085– The maximum size alphanumeric output was reached. Contact your BusinessObjects administrator. (Error: ERR_WIS_30272)
- SAP Note 1820378– Error: “Maximum binary output size limit reached. Contact your Business objects administrator (Error: ERR_WIS_30271) (Error: INF)” while saving WebI as Excel or PDF
- SAP Note 2025914– Maximum Character Stream Size” setting affects the export of Web Intelligence reports to xlsx format (not the case with the other formats)
- Impact of increasing the “Maximum Character Stream Size (MB)” value on the Webi Processing server? via SAP Community Network
Other Resources
- Office Open XML from Wikipedia
- SAP KB 2009536– “Could not load the Selected universe : An internal error occurred while calling ‘getSessionInfosEx’ API. (Error: ERR_WIS_30270)”
- SAP Lumira and (Insert Witty Pun Here) the Universe by Ashish Morzaria on SAP Community Network (added October 8, 2014)
Great article! What exactly is a medium, large and extra large data set in the Lumira world?
Ryan,
I don’t know what exactly medium, large and extra large data sets are, other than I have data set envy. We were having issues yesterday with some perfectly reasonable queries that we’ll be analyzing in Web Intelligence so we can validate the SQL. There is some evidence that patching up may also alleviate the issue, so my customer is also moving up from BI 4.1 SP2 to SP4, which will have the added benefit of resolving several other issues that we were mostly able to work around during their migration project. My customer is not a SAP business suite customer, so HANA is just not a realistic option. They need to be able to query the data from where it is, which is in an Oracle database fronted with multiple existing universes. The universes are supplied and supported by a vendor and not homegrown, so changing them isn’t an option even if there’s a way to make them better (for example, if the measure objects don’t have aggregate functions or some other low-hanging fruit opportunity).
Regards,
Dallas
HI Dallas
Great article and you will be pleased to know that SAP Lumira team is working on this very requirement.
Thanks
Sharon
Sharon,
That’s great to hear. Customers know that the Lumira team is working at maximum speed and we appreciate the frequent agile updates. This article sat unfinished in draft mode for several weeks but literally was my life story yesterday as I helped a SAP customer who is on the front lines of the Tableau battle.
There’s an episode of Sherlock with Benedict Cumberbatch and Martin Freeman where Sherlock tells John to not blog about the unsolved cases but John replies that “people want to know you’re human.” I felt very human yesterday telling the customer to “keep increasing the value until the error goes away”. And of course we had to build queries in Web Intelligence to use the “View SQL” button because there is no such button in Lumira to help a user understand why they can’t get their data set back.
The plus side is that there ARE organizations out there trying to make SAP Lumira work in their environment. The negative side is that new employees are joining the organization saying “I had Tableau at my old job and would like it here.”
If I’ve missed something and there is a more deterministic way to appropriately set the maximum character stream other than guessing (and without making the rest of the system sluggish because it’s set too large), please let us know. With regard to query panel interface, hopefully because of their shared Eclipse foundation there can quickly be more parity not only with SAP Lumira and Design Studio, but also with the gold standard of Web Intelligence (whose creators are free to innovate to keep making even better, BTW).
Thanks for writing,
Dallas
Good catch, Universe integration uses webi sdk, and good news from Sharon that development teams are working on a better solution.
But other than a good Query panel, i believe we should be able to:
use pre-defined queries at universe level.
work “server-free” in agile scenarios “Local universes”
For that, i published my idea here if you’d like to check and vote
https://ideas.sap.com/SAPLumira/D20681
Keep on truckin Mr Marks! If they listen to you then maybe, just maybe, by the time my clients are ready for Lumira (late 2015) it will be ready for them.
Thanks, Deanna. It was very good to see you again at SABOUC in Fort Worth last week.
We’ve found you also need to increase that Maximum Character Stream Size if you are using Explorer based on UNV format universes. It must use the same mechanism when indexing those Information Spaces as it does in Lumira.
Ryan, thanks for reminding me about Explorer. Looks like I wrote about it’s use of maximum character stream size last year and I’ve updated this article with a link. Kind of ironic that Web Intelligence is rarely mentioned but its plumbing is critical to so many other tools in the suite.