Back when SOA first started getting traction, the goal was simply to make application functionality available as a shared service. Companies made up their architectures as they went along — and of course, they’re still doing that. The difference today is that, in the last couple of years, the business side has a better sense of the strategic value of IT, while IT has learned more about the competitive pressures business must endure. As a result, SOA now offers the possibility of greater alignment between IT and business than ever before.
What business needs is an array of services that can be recombined, resulting in new business processes that support new products or services. Publishing those services and providing a coherent framework within which they can be governed and orchestrated into applications is what SOA is all about. Although many SOA initiatives remain an early phase, the promise of increased business responsiveness is real. And we’re seeing an increasing number of enterprises that are pushing toward advanced deployments. The case studies here are prime examples.
Comcast builds its SOA on domain expertise
It’s tempting to start buying ESB, registries, and other tools when you decide to adopt the SOA approach. But that misses the key value of SOA, which is to have a way to align the applications you create and deploy to the business processes they execute, says Tom Adler, senior manager of application architecture and IT governance at Comcast. And starting with the architecture can help ensure you have the right framework to do that — both now and as needs change over time, he says.
“When we started this effort 18 months ago, we resisted the temptation of bringing in vendors. We brought in subject matter experts instead and figured out what we needed first,” Adler says. “All the vendors just wanted to sell us an ESB.” The architecture effort does more than set the framework, he notes: It also begins identifying where you already have redundancies, both in business processes and in applications. That’s key to getting business buy-in, Adler says, because it shows in very real terms where there are opportunities for savings that will help justify the eventual SOA infrastructure and tool investment, as well as show where the use of common services should help reduce maintenance and integration complexities, allowing more responsive development efforts in the future. “It’s the target that lets us eliminate redundant services,” he says.
After developing the architecture — what Adler calls the common domain model — Comcast’s next step was to develop the governance framework for the service development and deployment. “Services need to go through the governance gate,” he says, “otherwise no one will know that the services exist or follow the right policies and procedures.” Only services that pass the governance gate are added to the service registry and thus made available for others to reuse.
One governance challenge that came up quickly was deciding who owned the services. Comcast is fairly decentralized, so the culture naturally supported having the service originators own the services in their domains, Adler says. Common services, such as single sign on, reside with IT — their natural domain, he adds.
One step that Adler now realizes Comcast missed was developing a common data service model after defining the architecture. By not having standard data services to access corporate information and manage interactions across systems, developers have ended up designing their services to get the job done in different ways, leading to inconsistencies that break the SOA promise of allowing an easy mix and match of service components. “We underestimated the value up front,” Adler says, and the price has been reworking some services to impose that model after that fact.
The architectural focus of Comcast’s SOA effort has helped the concept be applied more widely than if viewed merely as a technology issue, Adler believes. For example, because Comcast didn’t start with the view that SOA means the use of Web services, the company has applied the SOA concept to all its efforts, not just those that are obviously Web-enabled. “A Web service is just one way to expose a service — it’s just an implementation detail,” he says. One result is that much of the initial internal SOA efforts were in fact directed at the legacy applications, reducing the integration points both within and outside the company (such as with billing vendors), a major pain point for the business.
Developers use a variety of tools and programming languages, for example, as best fit their knowledge and the application they are creating. By standardizing processes and policies instead of specific tools and technical methods, Adler says developers can better adhere to the architecture’s intent rather than trying to shoehorn each effort into the limitations or assumptions of a specific tool or technology. There’s also a practical reason to allow technological heterogeneity under the common architecture, he notes: In a 9,000-person company, it would be unrealistic to get everyone on the same methodology.”
A company of that scale must also adjust to changing business needs and technology opportunities, Adler says. It’s important to revisit the reference architecture periodically so it doesn’t become a straitjacket or a document that everyone ignores; either way, you would lose the benefits of SOA. Adler revisits the architecture each month, though it is changed less often than that.
Leapfrog keeps its SOA options open with open source
It’s the classic problem faced by IT developers everywhere: Applications developed during the year by different groups just don’t work well together when brought into a common system such as a Web portal. The dilemma hit home for Leapfrog Enterprises in early 2007 when the toy company wanted to deliver its various applications in a consistent way to suppliers and customers alike, wanting to take better advantage of Internet-based commerce and transactions. In March, it decided it needed a new way to develop applications, so it began an SOA effort whose first efforts are now bearing fruit, says Eugene Ciurana, the director of systems infrastructure. “We wanted to lay the foundation for Web infrastructure and systems, so we stated with a clean slate,” he says.
Leapfrog had many of the same goals that typify a typical SOA initiative: greater reuse of code, faster development time, and easier integration. But the company did not want to approach SOA as simply a changing of the guard for development tools and integration platforms. Instead it wanted to free its developers from conforming to a platform’s idea of best practices so they could focus on the applications’ functionality and use a wide range of development technologies as best for each job. (Leapfrog’s developers use a mélange of Java 5 and 6, Microsoft’s C#, and Web services with various third-party libraries.)
For example, Ciurana did not want to force developers to all use the same transport. “The transport doesn’t matter,” he says. He chose to use the open source Mule ESB as a messaging backbone, relying on it to deal with transport interfaces. That way, “developers could focus as little as possible on the implementation of services,” he explains. Instead, their focus is on the functionality they are trying to achieve. The result is that developers tend to use HTTP as their transport mechanism, but some use REST (Representational State Transfer) and SOAP — “whatever works best or they’re most comfortable in,” he says. With the Mule ESB’s approach, “they don’t need to worry about what’s in a specific SOAP stack or what IDE they’re using.” Ciurana had previously used Mule at Walmart.com, so he was convinced it was the right basis for Leapfrog’s “clean slate” efforts.
Leapfrog could take this approach because its focus is on integrating applications, Ciurana notes. “Most of the integration is happening at the app level, with apps talking to apps. So apps just do inputs and outputs,” he says. Developers write services as POJOs (plain old Java objects) and let the Mule ESB “wire” the POJOs to the messaging network, handling any transformation within the ESB. “Normally, when you’re working with SOAP and REST, you tend to think about how to connect to the outside world. With POJOs, you don’t need to do that,” he says.
Ciurana also likes the simplicity of the Mule ESB because it has no agenda other than to manage the messaging. “All the commercial vendors wanted to sell us a whole suite of products. Yet the whole point of SOA is to move from one locked-in system to another,” Ciurana adds. With the Mule ESB, Leapfrog has to assemble the various layers of the SOA stack, but Ciurana is happy to pay that price to have the flexibility to use whatever works best at the time.
Leapfrog uses two ESBs, one for managing data flow and application handoffs throughout internal systems such as ERP, ActiveDirectory, and data warehousing, and the other ESB for customer-facing Web-based applications, such as its customer account self-management application and the online games it offers consumers. Not only does this bring in a natural boundary for security and access management, it also provides a mutual backup capability, as each ESB can take over for the other if needed.
Leapfrog did have to create a common service-naming scheme so services could run on either ESB, Ciurana notes. The exacting names “help us keep all the services straight” he adds. It’s a small price to pay for ESB freedom.
United Airlines merges SOA with event-driven architecture
As sensible as the SOA concept is — decomposing business processes into their constituent elements and then developing stand-alone software services that let you mix and match the components as needed for a variety of business demands — it assumes that you’re dealing with discrete transactional functions. The basic SOA premise requires that function be building blocks that can be combined in nearly unlimited ways.
But many business tasks are not so decomposable, instead relying on specific sequences of events. Airlines are a classic example of an event-driven set of processes, and so they typically have an EDA (event-driven architecture) in place to handle events. “EDA is very flow-oriented, while SOA is about discrete black boxes,” says Ramnath Cidambi, middleware engineering manager at United Airlines. Both have their place; after all, airlines also have transaction systems such as booking fares and assigning seats, not just event-driven ones such as dispatching fuel trucks when a plane lands or updating the status board for flight arrivals, he notes.
United has long invested in its EDA, using IBM’s WebSphere as its messaging bus for seven years. In the past year, it’s begun an SOA effort to handle the modern Web services used at its United.com Web site. But those two environments are fairly distinct, Cidambi notes, so they could exist in parallel. However, that’s beginning to change, as the company begins adding transaction services within its internal operations, such as notifying customer service reps by text message (using Web services) what their schedules are for the day, using the HR system to see who’s scheduled, who’s called in sick, and so on to assign individuals to various gates at each airport. That puts Web services into the same environment as event-driven processes, Cidambi says, causing the airline to begin an SOA effort beyond the United.com program.
United’s challenge is figuring out how to architect and deploy services in a business that has, and needs, two architectures. Although the airline’s internal operations have two architectures, it cannot treat them as completely separate. After all, a cancelled flight (an event) also has implications for transaction systems (such as rescheduling passenger flights, updating Web-based flight-status lookup tools, or issuing credit vouchers). Many processes have both an event and a transaction component: while customer service reps get their schedules for the day from the transaction system, changes in plane status due to cancellations, weather delays, etc.quickly render that schedule moot. The event-driven system tracks the plane status, and the scheduling transaction system then updates the staff’s assignments by polling that status periodically. (The flight display monitors use the same process.)
The biggest challenge has been figuring out the messaging system. “ESBs don’t use standards outside of Web services standards,” Cidambi notes, so how to deal with event-driven services is unclear and inconsistent from product to product and tool to tool. Yet Cidambi is drawn to using ESBs for both SOA and EDA purposes because they handle messaging, data transformations, and other critical data routing tasks well.
Today, United Airlines has two ESBs, one for EDA services and one for SOA services. It uses an IBM WebSphere integration broker as a publish-and-subscribe-oriented messaging platform for its event-driven services, propagating events as needed and handling any transformations among services — essentially acting as an EDA ESB. For the transport, its existing J2EE applications are very messaging-oriented, so all use JMS (Java Message Service) as the messaging standard rather than Web services.
United is adopting the BEA AquaLogic ESB for its SOA services, because it is a newer platform that Cidambi expects will be more oriented to the newer SOA concept, and a better fit with the WebLogic server environment and Eclipse development environment in use at United. “AquaLogic basically runs on top of WebLogic,” Cidambi says, so there’s no integration effort.
Avoiding unnecessary effort is also why Cidambi isn’t moving the EDA services to AquaLogic; WebSphere does the job quite well, he notes, and moving to a new ESB would inevitably cause disruptions and surprises: United has had seven years to optimize its WebSphere platform and work out operational kinks; moving to a new ESB such as AquaLogic would take effort now and undoubtedly expose new kinks.
Another issue Cidambi faces is the lack of standard XML schemas for EDA, making the messaging effort between EDA and SOA services more complex and labor intensive.
Thomson Financial automates service compliance
Many companies like the SOA concept because it promises faster development times. But some SOA developers have discovered that a key part of service governance can actually slow down development, stealing that promised speed. Financial publisher and information services firm Thomson Financial discovered this unpleasant surprise early in its SOA journey, recalls Vladimir Mitevski, vice president of product management core services.
“To call it an enterprise production asset, a service needs to comply with several methodologies and policies,” Mitevski notes. Many are very exacting: The names of XML elements can’t be abbreviations and must be real dictionary words, for example, and some items, such as user names and passwords, cannot be hard-coded. When you have just a few services, the enterprise architecture team can usually keep up and also catch any issues, he says. But soon enough, the reviewers become a bottleneck and even begin missing problems due to the workload.
Thomas Financial has thousands of services — fine-grained, coarse-grained, and everything in between — and a small architecture staff, so it quickly felt the pain. “No matter what the granularity, each service goes through this process,” Mitevski says. Only then is it entered in the service registry. Likewise, changed services need to be evaluated for compliance before the new version is registered and thus available for production use. “But the architecture office was a bottleneck due to the scale,” Mitevski says.
Reducing the compliance requirements was not an option, given the critical nature of the applications involved, such as single sign-on services, Web services that deliver financial markets information to analysts and traders, and Web-based financial analysis and charting services accessed through Microsoft Office.
Thomson’s solution to the compliance workload issue was to turn to automation, using policy evaluation tools from WebLayers. “The tools are more efficient and don’t miss violations,” Mitevski says. It did take some time to create the policies that the tools gauge compliance against, and it’s critical that architects review the tools’ analyses to see if certain issues arise repeatedly that might indicate lack of understanding of key policies by developers or ambiguity in the architecture, he notes. “It helps show us what to do better, and some policies do need to be adjusted” Mitevski says, though he has found that most violations are due to developers taking shortcuts. Architects also decide when developers are granted exceptions for any compliance violations, something that happens only rarely, he notes, and the exceptions are noted in the registry for other users to be made aware of.
For Thomson Financial, the results of service compliance automation are dramatic, Mitevski says: “It used to take 20 people in a highly orchestrated process across various groups to go live [with a service]. Now it takes just a single person.”
Jabil Circuit simplifies customer integration
A company that focuses on manufacturing services needs to accommodate a wide range of customer integration, such as for billing, forecasting, and order systems across the many systems in use by its customers. But it’s very hard to manage all those point-to-point communications as your customer base grows and evolves its own systems. That’s why many manufacturers have moved to third-party providers of transaction hubs, called VANs (value-added networks); each supplier and customer worry just about the one connection to the VAN, and for each customer-supplier pair.
But this approach fails when you have custom processes with your customers that the standard VANs don’t accommodate. Jabil Circuit, a custom electronics manufacturer, faced this dilemma the hard way: by manually maintaining all those custom applications and interfaces. Jabil has more than 5,000 trading partners, though most could be handled using the VAN approach. But 50 customers needed special communications mechanisms or business processes that the Sterling Commerce VAN was designed for. Often there were several such custom connections per customer, compounding the effort, recalls Lowel Gilvin, the company’s e-commerce manager. Something had to change, so Jabil adopted SOA principles to replace most of these custom connections with service-based ones that let common functions be reused.
The first step was to separate the business processes — such as order-to-cash management, forecasting, and inventory consignment — from the communications processes. Jabil now has standard services for most of the communications mechanisms in use, such as AS1 (Applicability Statement 1), AS2 (Applicability Statement 2), and FTP, as well as separate data-handling services such as for XML, flat-file, Excel, and SAP iDocs formats. It composes the appropriate communication service, data-handling service, and business service for each of these customers, using tables and metadata to automate the composition in most cases. In some cases, customers use more than one mechanism, perhaps based on which division is involved, and the tables account for these multiple mechanisms, Gilvin notes.
The SOA concepts of abstraction, modularity, and service composition usually work as is, Gilvin notes. In some cases, special requirements can’t be met by combining services, so Jabil still has some one-off integrations to maintain. But even here, Jabil can often use the SOA approach for part of the integration. As one example, certificates for XML and SSL validation can’t be handled as standard services, since the certificates are unique, but Jabil can compose the appropriate communications and business services with a hard-wired data-handling service, keeping the reuse and consistency benefits of SOA in two of the three integration aspects, Gilvin says.
Rather than use an ESB to manage the messaging, a registry to manage the services repository, or a SOA-oriented development environment to develop the services, Jabil uses Sterling Commerce’s Gentran Integration Suite for all three purposes. The suite is designed for supply-chain interactions, which is all that Jabil is trying to manage. This limited scope lets Jabil rely on the toolset’s embedded architecture rather than create its own. “We have a small set of standard business processes,” Gilvin notes.
