Lead programmer at Streamsim Technologies for studioSL. I have a degree in Computer Science from the University of Padova, Italy, with a thesis on a GUI of an unmanned vehicle written on top of NetBeans Platform. Matteo has posted 1 posts at DZone. View Full User Profile

Reservoir Engineering on the NetBeans Platform

02.15.2011
| 5928 views |
  • submit to reddit

I'm Matteo Di Giovinazzo, lead programmer at Streamsim Technologies for studioSL. I have a degree in Computer Science from the University of Padova, Italy, with a thesis on a GUI of an unmanned vehicle written on top of the NetBeans Platform.

In a Nutshell: studioSL

studioSL, by Streamsim, is a graphical user interface to allow engineers to pre/post process simulation models of oil, gas, and water flow in the subsurface as well as monitor the simulations in progress. For example, here's a screenshot with data clustering visualizations:

Initially, studioSL was focused on Streamsim's own streamline-based flow simulator, 3DSL, but we have gradually introduced a common API to access and process simulation results from other simulators.

Using studioSL, a reservoir engineer is able to create, edit, and manipulate an input data file, launch 3DSL or another simulator, and interpret the simulation results. In addition, the standard 2D and 3D plots that are generally accepted in the industry as good metrics for deciding the usefulness of the output. However, because 3DSL produces some very unique data, studioSL also produces unique 2D and 3D plots:

The ability to graphically display this data was one of the original reasons we decided to venture into building our own pre/post processing capabilities.

More recently, we are working hard to create workflows that guide engineers, through creation of sets of interdependent runs, to improve their reservoir models. These workflows are central in allowing engineers to make better decisions. Examples of workflows we currently have in studioSL are:

  • improved waterflood management using floodOPT:

  • streamline-based assisted history matching (model calibration):

  • ensemble analysis (use many simulations to enable a statistical analysis of the model response):

     

Target Audience of studioSL

Our typical user is a reservoir or exploitation engineer in small to intermediate oil companies all the way to large major international oil companies and national oil companies. These engineers might be responsible for multiple fields and multiple wells and for well plans for improving declining recovery. We also have a large number of clients using our software for research purposes.

Here you see how data and images can be exported:

History of studioSL

studioSL started as a very simple Java Swing-based XY plotting package to visualize data from 3DSL in 1999. Increasing functionality in 3DSL, requests for more features by clients, and the need for a platform we could control (vs hooking up with already existing pre/post software) forced us to consider designing a more complex software, at which point we decided to convert it into a NetBeans Platform application:

The rest of the studioSL team is as follows:

  • Marco Thiele: co-founder and president of Streamsim Technologies. He has a PhD in Petroleum Engineering from Stanford University and share programming responsibility for 3DSL, the numerical flow simulation engine for which studioSL was developed. His main responsibility is to understand how exactly our clients might interface with our software, tries to bring innovation into Streamsim products in the form of new workflows to help engineers make better and timely decisions. He also does consulting, teaching, and presentation of the technology at conferences and workshops.

  • Rod Batycky: co-founder of Streamsim Technologies and developer of 3DSL. He has a PhD in petroleum engineering from Stanford University with major interests in reservoir simulation. He does some programming of our simulator, simulation consulting projects with clients, and guiding development of studioSL.

  • Darryl Fenwick: reservoir engineer and project leader of Streamsim/Stanford Joint Industry Project for history matching. He has a PhD in petroleum engineering from Stanford University. His major interests are in reservoir model calibration to historical production data, and quantification of uncertainty in future reservoir production. He programs in studioSL, writes custom build scripts in Ant, and geostatistical algorithms for Streamsim applications.

  • Paolo Repele: developer at Streamsim Technologies, Inc. Degree in Computer Science in 2007 in Padova (Italy) with a thesis about a Modular License Manager integration in the NetBeans Platform 5.5.1.

  • Enrico Scantamburlo: programmer for studioSL where he looks after Schliemann, MultiView and Office integration. He has have a Computer Science degree from the University of Padova, Italy. He also has a SCJP programmer certification.


Technical Challenges

Scalability is a major issue we are confronting. Over the past years, we've faced significant technical programming problems, mainly because of the growth in size of typical reservoir models. The ability to be able to visualize large data sets (100's of MBs of data) using the same infrastructure we use for smaller models is proving to be an issue.

Lately we have also experienced problems handling large sets of files. For example, we might create 1000's of simulation models (input files), with one simulation input file (and subsequent output files) per folder. However, we must also support cases where there are multiple runs (input and output files) in a single folder:

Guiding the user through the creations of those runs is the best way to go, offering a complete workflow project to organize step by step creation, execution and analysis of thousands of runs. Thanks to the NetBeans Project API, we also provide a logical view (seen below in the leftmost pane) hiding the most cumbersome details of a fine grained physical file system view:



We also have to support a network installation of studioSL. In fact our clients work almost exclusively with network environments, and often install studioSL on a network in order to allow users to run our application from a generic workstation with the same user data (opened projects, settings, etc.).

We experienced huge differences between Windows clients and Linux clients in terms of performance over the network. For example, listing and filtering files of a large folder in network is really a slow task in Windows clients. We used this trick: instead of using File.listFiles(...) or FileObject.getChildren() methods and then filtering for 3DSL files to be shown in out project view, we launch a "DIR *.3DSL" command and ask for File/FileObject only for visible nodes. We think this is related to Windows client (not server) and not to Java/NetBeans. In future we might take advantage of JDK7 NIO2 that offer support for directory streams.

In the future, we also want to support remote execution of simulations in a server cluster and be able to queue jobs when the number of runs to manage is big. In other words, while studioSL might be running on client A, the actual numerical simulations need to be spawned off to a cluster, potentially queued either in a specific way or reusing an existing queue management infrastructure like Sun Grid Engine.

External execution of a simulator (3DSL or other) and capturing the output in the NetBeans Output Window as well as synchronization of logical and file view of the application is an issue for us.  For example, if 3DSL creates thousand output files from a single input file, studioSL needs to be notified of each file creation (and subsequent update), slowing down the performance. We solved this with a few hacks, but we are still not completely happy with the result.

As clients using fast clusters to run their simulations and store data on network drives, studioSL will have to be able to access large data sets across the intranet efficiently.

Deciding to Use the NetBeans Platform

Initially the decision was between NetBeans and Eclipse. We decided to go with NetBeans because as a very small company (we still are) we needed a platform that allowed us to build an infrastructure cost effectively and to leverage existing technology without us having to reinvent. At the time, Eclipse needed to be licensed and was expensive. We also wanted to ensure the ability to run on multiple platforms, although at this point only two have survived for us: Windows and Linux.

Also, NetBeans is open source and free, giving us the possibility of contribution to improve it when needed but also rely on the support of the wider community. In a world where most all software is proprietary, the NetBeans open source status seems to be well received. There is also an active community source of ideas and continuous training for all our developer team.

Benefits of the NetBeans Platform

  • Free and platform independent

  • Allows us to concentrate on developments that are specific to our needs without having to reinvent all the standard infrastructure. The update functionality on NetBeans is a good example of this.

  • NetBeans offers a good way of designing API, following Lookup and inversion of control guidelines.

  • Using XML MultiView has given us a UI to easily modify XML, without having to handle the complexity underneath.

  • The use of Schliemann had saved us from writing our own parser for the 3DSL syntax. It has also given us an object model to create a UI based on it.

  • We implemented several projects and workflows using NetBeans Project API and ANT-based Project API 


When we started, there was no NetBeans module creation facility - we managed to create the build infrastructure for each module and integrate them in a RCP application.  Subsequent improvements in this area by NetBeans have been of significant benefit to us.

Every NetBeans release comes with new features and new improvements. For example, as soon as the exception reporter was available, we reused it to enable us to send bug reports and statistics information to our specific issue tracker:



Sometimes a new feature in NetBeans leaded us developers to propose new features in studioSL, like with Schliemann. Such an easy way to plug-in the syntax highlighting made possible to open our input simulation file with colored keywords, folded sections, and syntax auto-completion.

Places Where the NetBeans Platform Can Improve

  • One big missing feature is to upgrade from a release of NetBeans Platform to the next via plug-ins updates. We tried this several times (from 6.0 to 6.1 and from 6.1 to 6.5) but without luck. Unfortunately studioSL is still based on NetBeans Platform 6.5: the main reason to be so out-dated is that we would want to send fixes and updates through NBMs using Autoupdate Services, not forcing our users to download and install a new .zip every time. At each new release of NetBeans we evaluate if there is some new features that force us to jump to the next NetBeans Platform version. We, as developers, have had a hard time to remain based on NetBeans Platform 6.5, since we noticed a lot of new interesting APIs and features in the last NetBeans releases. Recently we have been testing the Remote Execution API (plugin under development contained in the C/C++ cluster) to use it in our remote job launcher and we are going to release a new studioSL based on NetBeans 6.9.

  • We also deeply test and usually wait to adopt the latest NB version since in the past we found it risky. For example, with NetBeans 5.5.1 we extensively used MultiDataLoader, since our main DataObject (3DSL DataObject) is constituted by a primary file that is the input simulation file and a large set of output files, this way in every file view we show only a single node, hiding all output files behind it. When we switched to NetBeans 6.0, there was a regression: we had very bad performance with MultiDataLoader. This forced us to change our architecture handling the DataObject with an UniDataLoader and take care of hiding the secondary files only in the Project view implementing by ourself the logical view of our projects, instead of just delegate to the file view. Here comes the trick I mentioned above, about listing and filtering files in large folders: under Windows when we create the logical view of a project, we actually call "DIR *.3DSL" in order to show only .3dsl files (and instantiate DataObject and DataNode only for those files).

  • FileSystem API issues with external executions and file creation/change/deletion. This is really important for us: in some project implementation we completely depend on those events to update our logical view. Moreover some operation is done via ANT scripts, that requires updates of the logical view in the middle of the execution (for example we create a bunch of simulations and we need to show them to the user, even if the creation ANT script has not been completed). To do so we use some custom tasks that creates files and folders using FileObject methods, instead of File methods. This way the NetBeans FileObject cache is always up-to-date.

  • Another point about performance in our current architecture is the population of DataObject's Lookup. We managed in this way: every 3DSL DataObject contains data depending on existence of output files with the same name in the same folder of the main simulation file. In the  DataObject constructor (and every time we receive a file change event) we change the content of the DataObject's lookup, adding or removing "data providers". Those providers are therefore used in the actions off of the 3DSL node, like "Plot Well Responses" or "3D Viewer". This I/O to ask for files takes time and we are trying to postpone it in the performAction() method instead of doing it at DataObject creation, but this is not always simple. We would like to enable/disable the actions depending on availability of some data provider from the current node selection (or lookup context).

  • Project Groups or relationships among projects. Now that we have a lot of projects and workflows in studioSL we would like to join them in a single entity. Maybe "project groups" can fit this idea, but there is a lack of relationships among them. We need a way to link a project to another and a set of project to a single main project. We are envisioning an Ensemble project that reuses pieces of existing project coordinating them in a single multi-project workflow.

  • Installing Executables for the right OS and architecture. We ship several executables with studioSL. Since we support Windows and Linux both in 32 and 64bit, we need to ship four versions for each executable. We wrap each executable into a module to install only the correct one. The problem is that the NetBeans Platform currently does not support tokens for 64 or 32 bit architecture, so we need check by hand in the module installer.

Finally, here is a screenshot showing the set up of a simple grid containing all the well perforations:


What are some of the new features you're planning for future releases?

  • Remote launching of applications.

  • See Project Groups here above.
Published at DZone with permission of its author, Matteo Di Giovinazzo.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Geertjan Wielenga replied on Tue, 2011/02/15 - 6:33am

That's a really beautiful app, thanks for sharing this info.

Next, your need for project groupings, i.e., different levels of projects that relate to each other, can be handled via the org.netbeans.spi.project.SubprojectProvider class. You first need to create a main project and make it pluggable, as described here. Then read this to use the SubprojectProvider class. It's interesting, but there's a reservoir-related organization that I know of that's using exactly the same functionality to create project groupings in their NetBeans Platform application.

Matteo Di Giovinazzo replied on Wed, 2011/02/16 - 5:34am in response to: Geertjan Wielenga

Interesting suggestion about subprojects, but really the relationships we have are among projects as peers not as parent-child: the idea is that there is a whole series of projects all related to a single reservoir. For example there is a base case simulation (the starting point), then a project doing history matching (using the base case to create several lists of new simulations), then another project created picking one of the result simulation and so on. Then what we envision is a network of relationships among projects, for sure Visual Library can help drawing (and building?) such network visually. That network is useful for users to keep track of their activity, since the whole workflow can be complex and a lot branched. Do you remember the reservoir-related organization that is doing this with project groups?

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.