Andreas Stefik (pictured, right) is an instructor currently teaching at Central Washington University (CWU). He is beginning teaching as an assistant professor next year at Southern Illinois University Edwardsville (SIUE). His primary area of interest and research is in developing custom programming languages and compilers.
A few years ago, while he was searching for a dissertation topic, he became interested in a small community of highly talented computer programmers who are blind. He was curious how they went about completing their every day programming duties.
Below follows an interview with him around the topic of programming tools for blind programmers and solutions created on the NetBeans Platform by Andreas and students at CWU.
Andreas, how did you become interested in creating better tools for blind programmers?
U.S. census data indicates that individuals with sensory disabilities are at a significant disadvantage in the American workforce. Using existing IDEs, for example, is often incredibly difficult for blind individuals, making it difficult to find work in an already competitive market. On average, blind individuals live in greater poverty, are less likely to be in the workforce, and make less median income.
I was curious whether computer programming was feasible for a blind individual, given the correct tools, and set out to try and create better tools that could give some of these users the potential at a high paying career. I've been working for a few years now creating and testing programming tools for the blind that, hopefully, will make them more accessible. These tools are very complicated, as they involve custom compilers, debuggers, and sound architectures built into standard IDEs like NetBeans or Visual Studio.
This interest intersects with the NetBeans Platform. You have even been teaching NetBeans Platform usage to your students. How did all that come about?
Designing a curriculum and choosing what to teach in computer science at a university is tricky business. As you can probably guess, this is largely because the number of technologies is truly daunting! Worse, there's no magic guru that tells professors which technologies are important, let alone which ones we should dedicate class time on. Professors often debate these things endlessly, which is probably quite a healthy way to approach the problem. At CWU, and I imagine most places, we try to focus on the concepts, not so much the architecture for a particular platform, like NetBeans, Visual Studio, or Eclipse.
With that said, several years back we made the change to a Java based curriculum (from C++), and part of that switch was changing to a Java based IDE. We use jGrasp for our initial classes and NetBeans IDE for several more. For most of these classes, though, we are too busy focusing on the details of algorithms or data structures to focus on the details of the NetBeans Platform.
But how I got involved in the NetBeans Platform is related to my blind programming project. Every year, we have a course at CWU on software engineering, which lasts about 20 weeks total: 2 academic quarters. In this course, we send messages to local community members requesting project proposals and choose the ones that we think are about the right difficulty for the particular students that year. Students then submit resumes, bid on the various proposals, and we assign collaborative teams.
In the past, the tools that I've been working on for the blind have been largely home grown compiler and debugger architectures that I wrote myself in my spare time. This worked great for running experiments and doing research on the effectiveness of the environments, but it isn't particularly practical. Thus, this year I decided to take the plunge and rewrite my architecture, as part of our software engineering course, on the NetBeans Platform.
Here is a screen capture of the environment as a whole, annotated with details of what we have added. Notice that, while the screen capture looks just like NetBeans, that we have added a new programming language (Hop). In addition, our debugger is “omniscient,” which means that you can execute code forward and backward. For example, in most debuggers, you can click continue (the green button) and step the code to the next breakpoint. In ours, you can also rewind (the blue button) to the previous breakpoint. Similarly, most IDE’s let you step over a line of code. Ours also lets you step back.
This project was started with a team of four students: Brock Dunda, Andrew Haywood, Shazada Mansoor, and Daniel Garcia and together we created an environment called "Sodbeans" or the "Sonified Omniscient Debugger in NetBeans." The students, and myself, dove in, heads first, into the NetBeans architecture with no experience. The students did a truly fantastic job.What have your experiences using the NetBeans Platform been like?
When we started this part of the blind programming project, all of the students working on it had used the NetBeans IDE, but none of us had more than very basic knowledge of the NetBeans architecture. I suppose I should qualify; I've known Java for a very long time, over a decade, so I have a pretty good working knowledge of the language and API's. Knowing Java APIs, though, isn't necessarily helpful for learning the NetBeans APIs. So, to answer your question, I would say that I started learning the NetBeans Platform around September 2008. I've learned quite a bit over the last year, but I'm hardly an expert.
Learning the NetBeans Platform is not as easy as learning Java or another programming language. Sometimes online tutorials give us a hint of what we need to know to complete whatever we are working on at the time, but more often than not we would get stuck and end up on the mailing lists. Like all mailing lists, they are hit and miss, but are much better than nothing. I also think it's pretty important to work in teams on projects when learning an API as complicated as NetBeans. That way when (not if) you get stuck, you can ask a friend.
What are the main pieces that your custom tool for blind programmers provides?
Our custom tool incorporate tools for our 1) custom compiler, 2) custom text-to-speech engines, 3) custom debugger architecture, and a number of other minor components,
Let's now talk about the main conceptual stumbling blocks you had while beginning to work with the NetBeans Platform.
First, I should say that the NetBeans Platform is probably the best of the major IDE platforms I’ve used. At this point, I’ve used the Visual Studio, NetBeans, and to a much more limited extent, the Eclipse platforms for extending the respective IDEs. In my view, the NetBeans Platform is probably the cleanest, and best documented, of the bunch. With that said, any IDE is complicated and I think the three most difficult aspects of the NetBeans Platform to grasp were 1) lookup, 2) module dependencies, and 3) the specifics of the way the NetBeans Platform uses XML.Lookup brought our team to a screeching halt before we figured out how to use it. To be clear, the idea behind lookup is very simple: You “look something up.” This is obvious. Unfortunately, what is rather not obvious from any of the existing documentation is: 1) what you lookup, 2) when you need to look things up, and 3) how you look something up, and finally 4) Why you need to look something up.
Let me give a very specific example. We wanted to write our own custom debugger and integrate into the NetBeans GUI. Ideally, we want it to function similarly to the debuggers for other languages, with some accessibility enhancements for blind folks. We also wanted our debugger to not get in the way of the debugger for other languages. In other words, we want our custom language to seamlessly integrate with the rest of the NetBeans languages, except that when ours is loaded it gives auditory feedback to the user in the way we specify (which has been thoroughly tested in formal statistical experiments).
Our first thought on how to approach the problem was to look for an API that connects to the debugger in NetBeans and either instrument it with information from our compiler or otherwise manipulate it. In normal Java applications, this usually means that you do an import, instantiate an object or reference some static object, and then manipulate the objects in some way. The point is, when trying to figure out “what to lookup” to manipulate the debugger, it was difficult for us to figure out how to go about finding what we needed to lookup and the context under which to do so. In the end, we did not take this approach, as our custom compiler and debugger ended up making its own toolbars and windows (suggested to us by some folks on the mailing list), and this approach has worked well, with a few gotchas.
But in terms of lookup, my point is that the tutorials on lookup often say that you pass a class to the lookup mechanism and it will return an instance of an object, but it's not easy to find what to lookup for any particular application (unless the application just happens to be mentioned in one of the tutorials). It would be nice if there was a resource that listed the big picture features of NetBeans and told the programmer which classes you lookup to do something useful (e.g., manipulating the built in debugger for a given language, making calls to other parts of the platform).
OK, that was lookup. Now what about dependencies?
Besides lookup and before you do much of anything in the NetBeans Platform, you also need to understand object dependencies. The strategy usually goes like this: 1) figure out which module you need to be dependent upon (if any), and 2) do a lookup for whatever it is you need. Compared to typical Java, modules in NetBeans need to set explicit dependencies in order to access the functionality from that module. Philosophically, this sounds like a good idea, because you don't have to worry about the internals of a module changing in the next version of the API.
Unfortunately, literally every single student that I've worked with on this and other projects in the NetBeans Platform has been completely confused by the module dependency rules. My hypothesis is that this is because the module dependencies are not “visually obvious” in NetBeans. These dependencies are buried deep within the properties of the module you are working on. They are critically important for getting NetBeans modules working, but are hardly obvious in the IDE.
With dependencies, we also kept running into something that NetBeans calls “Harness Errors.” Most of the information we were able to find on these errors came from online forums from developers that appeared as confused as we were, but the gist is that they would temporarily stop our project from compiling. In our case, unloading and reloading our dependencies usually seemed to magically fix the problems, although we never did quite figure out why they happened in the first place and never seemed to find a good resource or tutorial on them. This is just one of many problems that seemed to come up with dependencies throughout the project.
And your other concerns related to XML used by the NetBeans Platform.
In the NetBeans Platform, you write XML files (sometimes called layer files). By modifying the XML you can specify actions or other ways of interacting with the environment. Conceptually, this is extensible and makes sense. The documentation for the XML, however, is difficult to find. We spent many hours browsing for the right “magic keywords” that tell NetBeans what to do. There are many examples on the Internet on how to do this, but we found it difficult to find any kind of general documentation on what words were used in the XML and what context to use them in.
For example, say you want to add gutter actions to a custom editor (e.g., the graphic for a breakpoint). What do you change in the XML? Should you even be using XML or should you be using lookup? Do you need a module dependency before you alter the XML; before you do lookup? In terms of the XML, we put what is below into what’s called a layer file:
<attr name="originalFile" stringvalue="Actions/Debug/org-teambeans-sodbeans-actions-ToggleBreakpointAction.instance"/>
<attr name="position" intvalue="500"/>
If you look at the above, you need to determine that gutter actions are accessed by first specifying an XML filesystem tag, then a series of four tags labeled folder (Editor, text, x-hop, GlyphGutterActions). Then you specify a series of actions that will be taken for the gutter action. Using XML is easy, but it is very difficult to figure out which words NetBeans uses to describe something like gutter actions. Worse, the words change sometimes in new versions of the IDE! What would be really nice is if, when you type the characters "<folder =" that NetBeans populates intellisense with what word choices you can use and an explanation of what each of the choices does. For now, though, it just says "no suggestions."
Some of your last point should be overcome with the introduction of annotations in NetBeans Platform 6.7. Finally, what's the current status of the tooling for blind programmers?
I am definitely excited to play more with 6.7 and see what you folks have come up with! In terms of the tool, the students completed an alpha, demonstration ready, version of the tool in the spring. A few of the students have since graduated and are job hunting, but the project is being continued and has recently gone open source. We have a website and a sourceforge page. We also have a number of extremely talented, and experienced, blind computer programmers that help us in one way or another on the project. It's been a great experience and I’m looking forward to continue working, and learning about the NetBeans Platform.
Here's a link to our sourceforge page:
And here is a link to our webpage:
Thanks Andreas and keep us updated on progress with this useful extension to NetBeans IDE!