Free Hand SQL – Dallas Marks

Web Intelligence and Free-Hand SQL

SAP has put yet another nail in the coffin of Desktop Intelligence.

SAP has put yet another nail in the coffin of Desktop Intelligence with Free-Hand SQL in the recent release of Support Pack 6 for SAP BusinessObjects Business Intelligence 4.1. Steve Yemm has put together an excellent tutorial on the SAP Community Network (see related SCN article, Web Intelligence Free Hand SQL (FHSQL)/Stored Procedures in BI4.1 SP06). I’d like to show just a couple of extra formatting nuances not in Steve’s article.

Free-Hand SQL isn’t a silver bullet (see related article, Free-Hand SQL Isn’t Free), but it is a bullet. It can help query data that no existing universe can access. Web Intelligence now uses a workflow that will seem very familiar to Desktop Intelligence users.

I’m going to use the Web Intelligence Java applet and choose the new Free-Hand SQL option for creating a new document.

Next, I’ll choose a universe connection.

For my query, I’ll paste SQL from an eFashion query for Year, State, and Sales Revenue into the Query Script editor, essentially a large text box.

SELECT
Agg_yr_qt_rn_st_ln_ca_sr.Yr,
Agg_yr_qt_rn_st_ln_ca_sr.State,
sum(Agg_yr_qt_rn_st_ln_ca_sr.Sales_revenue)
FROM
Agg_yr_qt_rn_st_ln_ca_sr
GROUP BY
Agg_yr_qt_rn_st_ln_ca_sr.Yr,
Agg_yr_qt_rn_st_ln_ca_sr.State

The Query Script “editor” provides the same editing features as its Desktop Intelligence predecessor- none. However, it is possible to validate that the SQL you pasted from elsewhere is valid.

Notice that Web Intelligence inferred that the aggregate function SUM should be interpreted as a measure object. However, the object naming isn’t terribly creative.

That is why you’ll want to add aliases to your SQL statement.

You can manually rename objects, which is helpful for setting column headings. Here I changed Sales_Revenue into Sales Revenue.

And voila! The results are exactly what we expect. Except unlike data from the eFashion universe, measures aren’t well-formatted.

Simply right-click on any value in the Sales Revenue column and choose Format Number. It’s near the bottom of what seems to be the world’s longest right-click menu. Does anyone else hope that Web Intelligence 4.2 will have shorter right-click menus?

Choose the desired numeric format.

And there you have it, a Web Intelligence document that uses Free-Hand SQL.

Some additional observations. The new Free-Hand SQL is also available in the Web Intelligence Rich Client…

However, it is missing from the HTML panel.

UPDATE: The Free-hand SQL option now appears in the BI 4.2 SP4 HTML panel, where only the Analysis View query option is missing.

This fact isn’t surprising, since Excel, Analysis View, Text and Web Service options are also missing from the HTML panel. Let’s hope that SAP announces some clear plans on how it intends to bring the HTML panel to feature parity with its two peers (see related SAP blog, SAP BI 4.2 SP3: What’s New in Web Intelligence). Someday, I hope that new Web Intelligence features first appear in the HTML panel.

In addition to creating new Web Intelligence reports from Free-Hand SQL, this feature provides new capabilities to the Report Conversion Tool, which were actually introduced earlier in Support Pack 5. Prior to SP5, Desktop Intelligence documents with free-hand SQL were converted by placing the SQL into a derived table of a new universe (see related article, Retiring Desktop Intelligence Free-Hand SQL). This approach could become problematic when hundreds of Desktop Intelligence documents were spawning hundreds of new single-use universes. The Report Conversion Tool no longer needs to create a universe to successfully convert free-hand SQL documents.

Additional Resources

Web Intelligence Free Hand SQL (FHSQL)/Stored Procedures in BI4.1 SP06 by Steve Yemm on SAP Community Network

What are your plans for Web Intelligence Free-Hand SQL?

Free-Hand SQL Isn’t Free

Free-Hand SQL is a good short-term solution but a poor long-term solution.

Editor’s note: I’m a bit embarrassed that this article has languished in draft mode for almost two years as I try to clean out my backlog of unfinished blog articles. But it’s still a relevant topic to discuss.

With custom dimension grouping finally added to SAP BusinessObjects Business Intelligence BI 4.1 SP2, the functionality gap between legacy Desktop Intelligence and its successor Web Intelligence is largely closed. One large gap remains, although it’s a controversial gap in my view.

Three words- Free-Hand SQL.

Free-Hand SQL is a feature of legacy Desktop Intelligence that allows a report to be created from a hand-crafted SQL statement. It’s typically used when there’s not enough time to create a universe. A highly normalized data model is sometimes too complex to model generically in a universe and nobody wants to bother transforming it into a star schema. Web Intelligence presently doesn’t support Free-Hand SQL, but the Report Conversion Tool does an adequate job of converting existing Desktop Intelligence documents that use it to Web Intelligence (see related article, Retiring Desktop Intelligence Free-Hand SQL).

Looking forward to real-world customer story: Zurich Insurance, retiring DeskI. I want to know what to do about no Free Hand SQL #SBOUC

— Cindi Howson (@BIScorecard) September 10, 2012

The user interface for Free-Hand SQL is spartan, so the query is usually crafted elsewhere in a tool with better SQL editing features such as Microsoft Access, TOAD, or the SQL editor provided by the database vendor then simply pasted into a humble box in Desktop Intelligence.

It’s a pretty basic box, without any tools to help the Desktop Intelligence user fashion a well-written query. You can validate the final result, though.

My friend and SAP Mentor Greg Myers had this exchange a few years ago when Free-Hand SQL was first mentioned on the Web Intelligence product roadmap.

@dallasmarks Just say “NO” to Free-hand SQL. #dieDeskieDIE

— Greg Myers (@gpmyers) May 9, 2012

SAP’s Matthew Shaw writes eloquently on the SAP Community Network about the benefits of the semantic layer and drawbacks of Free-Hand SQL (see related article, Use of Semantic Layer over ‘free hand SQL’).

Personally I’m no fan of free hand SQL. My blog on why: http://t.co/HJ8uC5Iknm #AskSAP #BI roadmap @SAPAnalytics

— Matthew Shaw (@MattShaw_On_BI) November 14, 2013

Matthew elaborates on several valid technical considerations, but this one stands out.

There is no central control over the SQL with free hand SQL. Should the database change, corrections needs to be made, or improvements made, then each and every document containing that free hand SQL needs to be inspected and manually updated. Compare this to the semantic layer where one change is made and that change is automatically propagated to all related documents.

Matthew Shaw, Use of Semantic Layer over ‘free hand SQL’ on SAP Community Network

In other words, maintenance nightmare! Free-Hand SQL is difficult to manage. If the data model or reporting requirements change, the task of tracking down and updating the offending SQL is tedious and time consuming. Of course, the manager that told you to “just get it done” with Free-Hand SQL won’t be around when the maintenance requests come in- she put “agile BI project management” on her resume and now has a much better paying job than yours.

SAP no longer supports SAP BusinessObjects Enterprise XI R2. But unfortunately, it still supports the decade-old philosophy that “all features of Desktop Intelligence must be crammed (eventually) into Web Intelligence”. In my view, a better approach would be to ask “where is the best place in the platform to support users that need to write free-hand SQL” and “what features can we add to the platform to support the lifecycle of free-hand SQL”. There’s an opportunity here to share SQL query design-time features across multiple tools in the BI suite, possibly leveraging some of the existing query builder functionality in Crystal Reports.

Although it first appeared on the official roadmap nearly two years ago, Free-Hand SQL in Web Intelligence does not have a public timetable. It is not mentioned in the “What’s New” document for SAP BusinessObjects BI 4.1 Support Pack 3 expected to ship at the end of this month. Unless SAP changes its mind, Free-Hand SQL is coming to a future version of Web Intelligence (UPDATE: Free-Hand SQL was delivered in BusinessObjects BI 4.1 Support Pack 6 – see related article Web Intelligence and Free-Hand SQL). But if we must have Free-Hand SQL in Web Intelligence, can we at least have something more elegant than a big box?

How do you feel about the SAP product roadmap for Free-Hand SQL?

Retiring Desktop Intelligence Free-Hand SQL

It’s the end of the world as we know it – time to send Desktop Intelligence reports into retirement.

Historically, it’s easier to create a SQL query and slap it into a Desktop Intelligence report than build or augment a universe. There are two methods to do this. The first method is to use the New Report Wizard to create a free-hand SQL report instead of choosing a universe. The second method uses a universe to build an initial query which is further edited in the Edit SQL panel. The Report Conversion Tool (RCT) can handle both cases – let’s take a closer look at what happens. I’ve used SAP BusinessObjects Enterprise XI 3.1 SP3 to generate my examples. In either case, I must convert these reports to Web Intelligence if I want them to continue to exist in SAP BusinessObjects Business Intelligence 4.0.

Let’s first consider a free-hand SQL query. For a simple illustration, I have used the eFashion universe to build the query and pasted its SQL into a new Desktop Intelligence document.

The Report Conversion Tool will examine the document’s SQL and build a derived table universe (first introduced in BusinessObjects Enterprise XI R2) with a single derived SQL table that embeds the query. Here is what the derived table looks like in the universe.

For the next example, I used Desktop Intelligence to build a standard query with the eFashion universe. But I then modified the SQL using the Edit SQL dialog box in the query panel. Unfortunately, it’s not immediately obvious when you open the query panel that the query has been customized. For example, the query panel below appears to show a simple query with Year, State, and Sales Revenue.

However, when the Edit SQL dialog box is opened, it becomes clear that the query has been customized with a WHERE clause limiting the year to 2004. To maintain the custom SQL and prevent the report from reverting back to the universe-generated SQL, I clicked the Do not generate SQL before running box when I authored the query.

When the Report Conversion Tool is used to convert this document, it does not create a new universe. It continues to use eFashion but also customizes the Web Intelligence SQL. As with Desktop Intelligence, this customization is not obvious from the query panel.

However, as with Desktop Intelligence, the customized SQL is visible from the View SQL dialog.

The good news is that in both cases, the Report Conversion Tool was able to convert the Desktop Intelligence report to Web Intelligence. However, there is some bad news, particularly for the free-hand SQL report. First, the Report Conversion Tool creates a cryptically named universe and places it in the Report Conversion Tool universe folder. If your environment has lots of free-hand SQL reports, you are going to end up with a large collection of small universes. It’s a support nightmare – the BI equivalent of suburban sprawl. In addition, reports that contain prompts with lists of values (LOV) will generate some pretty nasty LOV queries from the derived table. Frequently, the performance of the LOV queries is sub-par.

Although the universe LOVs can be modified to run faster, you should be always ask yourself if a universe is truly needed anytime Designer is opened. You’ll be much better served by a smaller number of universes that know how to answer many business questions and not a multitude of universes that each power a single report. When I help customers with migrations, I prefer to perform an initial run of the Report Conversion Tool on all reports except the Free-Hand SQL reports (by leaving the “Convert reports containing free-hand SQL” box unchecked). The Report Conversion Tool will flag the Free Hand SQL reports as “not converted”. Then, a simple query on the RCT audit table will generate a list of these reports. Next, I’ll work with the customer to see if any of the reports can be retired or redesigned in Web Intelligence. Ideally, some of these reports can be recreated using a universe instead of one generated by the RCT.

If you decide that the new derived SQL universe must stay, take a moment to look at the WHERE clause of the derived table. Try to move as many restrictions as possible out of the universe and into the Web Intelligence report as Query Filters. Fewer restrictions will make the universe more generic and capable of answering more questions (and satisfying future report requests) than the original report. For commonly used restrictions, add predefined filters in the universe to make report creation easier.

If the odds are unlikely that nobody outside of IT will directly use the universe generated by the Report Conversion Tool, evaluate if it makes more sense to replace the free-hand SQL Desktop Intelligence document with a Crystal Report. Honestly, I’ve been surprised that Crystal Reports is barely mentioned by SAP when discussing Desktop Intelligence. There’s even talk of adding free-hand SQL to a future (post BI 4.0) release of Web Intelligence (see related article, Free-Hand SQL Isn’t Free). So perhaps my advice to choose Crystal Reports is ill-advised? Would love to hear everyone’s thoughts.

It’s the end of the world as we know it— time to send Desktop Intelligence reports into retirement. But thankfully, the Report Conversion Tool, even in XI R2 and XI 3.1, can help us reach our goal. Happy conversions!