Creating a Disclosure Plug-in

This past week in Glimmer, in addition to helping partners with their projects, I have been working on some more details of the self-disclosure implementation.

I have been adding disclosure messages for more actions, and I am currently struggling to deal with a problem with slider bars. When a user slides a bar, the context setting is modified at every movement of the bar; this means that my disclosures appear every time a foreground color changes slightly, causing over fifty pop-up disclosures with one move of a slider bar. To help fix this problem, I am trying to look more closely at signals, including where they are sent and caught, to see if I can find a “mouse-up” signal that emits when a user lets go off the mouse. If I can find such a signal, I can create a callback that connects the disclosure function with the slide bar “mouse-up” signal , so that the disclosure will display only when the user stops moving the slider bar, instead of at every slight movement.

In addition to these hacks, I have begun creating an actual GUI plug-in for the disclosures, as pop-ups are unseemly and annoying. I have successfully created a window that will open by selecting a “Disclosure” option from the Filters menu (which is the location of many GIMP plug-ins, including MediaScript).  My first struggle was to allow the plug-in to open without an open image in GIMP. I solved this, and am now working on two projects- designing a more polished window, and capturing signals in a buffer pane of the window, in order to catch disclosed actions.

My goal for the next week is to add several more disclosures through the current “hacking” solution in order to learn where more of the actions in the GUI are stored in the source code. Also, I want to figure out how these GUI actions are called with signals so that I may begin catching emitted signals with my plug-in. I also plan to create a more permanent and finished disclosure log by closely modeling the logging pane of the MediaScript plug-in.

Nyquist Plug-Ins Trudging Along

We’ve been making some progress in our scripting efforts with the Nyquist plug-in for Audacity. Though we’re easily confused by the syntax and explanations in the Nyquist manual, we get magical help when SamR comes to look at the code. Computers may indeed be stupid, sentient, and malicious, but at least they’re terrified of SamR — so they always start to behave whenever he comes near. Knowing this, I may be able to have a working soundscape using free stock sounds from FreeSound by the next week. It’ll be completely scripted, ladies and gentlemen. Be prepared to be amazed.

Though the side project is becoming more simple and straightforward, I’m only just beginning to understand the bulk of my main project. Right now I’m trying to find a way to bring up a simple, non-interactive window in Audacity. This should be a very simple step that will help me as we attempt to incorporate the Mediascript interface into the Nyquist plug-in. However, carrying it out is a challenge so far. Audacity is written in C++. I have a background in C and Java, so the language is not completely foreign — just enough that I keep getting stuck on tiny syntax questions that are taken for granted in tutorials.  Also, Jordan understands GTK (a great, simple toolkit for creating a GUI), but so far we’re unsure about compatibility with Audacity’s graphical toolkit. We’re trying to use Audacity’s built-in understanding of windows to create a simple pop-up, but right now it’s not working.

We’re also not sure how we can develop our plug-in. There are a few different ways we can find to implement a plug-in, but none seem ideal for our purposes. We want to be able to code in C, but the only C-language development tools for Audacity focus creating audio filters, not new program extensions. We tried to look at the Nyquist source code (all in Lisp!) but haven’t quite found what we’re looking for. In fact, that’s not surprising since we’re not sure yet what we’re looking for.

(Also, my apologies to Nora for dangling prepositions.)

Translations of a different kind

Today, I’ve been facing the occasionally frustrating task of translating old student projects into language that is compatible with the latest implementation of MediaScript.  Every semester, the CSC 151 class is assigned a final project: design a procedure, (image-series n width height), that produces an image of width and height based on n.  Every image with a different n value should be distinct, and the images should be reproducible.  Students have a variety of different ‘helper’ procedures that they can use, and are encouraged to design their own helper procedures as well.  Some of the requirements have changed over time – students are no longer allowed to call the ‘random’ procedure, since the images generated cannot be reliably reproduced – but the general idea has remained the same.

I’ve been working on projects from the fall of 2007, and for the most part MediaScript hasn’t changed much.  Syntax is generally the same, since we’re still using Scheme, but some procedures have different names.  For example, the context-set-x set of procedures (context-set-fgcolor, context-set-brush, etc.) used to be named envt.set-x.  For the most part, a simple find and replace can take care of these differences.  Procedures that now take (lambda (col row)…) used to take (lambda (pos)…).

Since this is student work, though, there’s always the question of whether something can be streamlined while still preserving the original design of the student programmer.  In certain cases, there’s also the issue of whether or not an image-series procedure actually fulfills the requirements of the project assignment.  Some laboriously constructed images-series procedures do not actually generate one thousand distinct images.  At that point, I have to decide whether to redesign the entire procedure to make it fit the criteria, or simply update it to current MediaScript standard and preserve the original programmer’s design.

I don’t have any pictures to share this week, but eventually I will build a website to house the various versions of code and selected images that it produces.  That’s the long-term goal, anyway.

Implementing Self-Disclosure “Hacks”

In order to produce self-disclosures, we need to capture actions in the GUI. To do this, I have explored how the GIMP represents actions. It does not store them internally, but rather functions handle events triggered in the UI. I am adding self-disclosure to these functions and will continue to learn more about signal handling to see if harnessing signals would be a better option. I am currently implementing a temporary hack to explore how the UI and internal functions interact. I have added g_message statements to functions, which create pop-up boxes that display Scheme scripts when actions are preformed in the GUI.

To add these messages, I found the relevant code by beginning with the GUI, as I want scripts to match GUI actions, specifically tracing how foreground color is set. So, I explored app/gui/gui.c, and traced through the methods trying to find how the foreground color is changed. Gui.c sets up many the UI signals (which may be useful if I decide to trace the tools with handlers) and also has a function called gui_get_foreground_func, which ends up calling gimp_get_user_context. This led me to app/core/gimpcontext.c, which provided much more information on how context settings were modified. In gimpcontext.c, there is a method context_set_foreground, which is a general method which many tools end up calling.

There are many design decisions that I have to make to add these pop-up self-disclosing messages to GIMP. I have to choose whether to place the disclosure method in the method for each specific tool, or for the general method which the tools end up calling (context_set_foreground). There are obvious benefits to adding the self-disclosure to the methods in gimpcontext: it is much more general and clear. Only one method needs to be written for each type of action (such as changing the foreground color), and it easy to modify and find the self-disclosing tools as they are all located in one central file. While this method is much more general, it has many drawbacks: many outside procedures call the context_set methods, such as during initialization as well as the MediaScript plug-in commands. We don’t want a script to be logged when it is called through the MediaScript, as it already has a built-in log. In addition, these self-disclosures tended to repeat messages multiple times for the same action, suggesting the context_set method is being called more than once.

Thus, I decided to place the self-disclosure in the individual methods for individual tools. These methods are found in app/widgets/gimptoolbox files. This means that there are many more cases and makes the code much less general. For example, to add self-disclosure for foreground changes, I do not simply have one method in context_set_foreground, but instead have separate g_message calls for changing the foreground color through the basic color selection dialogue, the text tool, the color picker (eyedropper) tool, as well as the swap and reset tools. This lets me tailor self-disclosure much more accurately: with the basic dialog, I can make sure that the message only prints when the user clicks to “ok” button, and then only if the user has actually changed the color and did not cancel the action (whereas before with the tool in gimpcontext, the pop-up would appear every time the color was modified slightly in the dialog, even if it was canceled).

Soon I will have to design a logging window to unobtrusively display a log of scripts that relate users’ actions to scheme syntax, as pop-up boxes can be very annoying. I should also consider how to make this tool optional and to generalize it to work with Python scripts. In addition, I plan to finish expanding self-disclosures for all of the context settings, and then work on messages for selections and for drawing basic images such as straight lines.

Updates for Nyquist

So, after a week of trying to make sense of the Nyquist Reference Manual and some attempts at scripting that were mostly guesswork, Jordan and I realized that there is one big problem with our current approach to Nyquist: this extension is used primarily by profession sound designers who script for ease, not for scripters who play with sound for fun.

We find ourselves constantly reminded of this as we pore over the explanations of Nyquist functions in the manual, only to realize that we have no idea what the thourough documentation means. Of course, we are grateful for any sort of documentation, but the Nyquist manual’s explanations assume a strong background in sound technology. Since we just jumped into Audacity and sound-editing in hopes of an interesting new scripting application, we have no such background.

Since Nyquist’s users are looking for the simplest way to achieve the best results for their audio, Nyquist’s interface is very minimalistic and utilitarian. There is a plain text box for inputting simple scripts, a debugging tool, and buttons for “OK” and “Cancel.” Though it works fine for quick scripting, we found this user interface to be inadequate from a programmer’s perspective. There is no user-friendliness in the form of syntax highlighting, definitions, or logging – all of which are equipped in our MediaScript plug-in for GIMP.

We don’t mean to say that Nyqist is a bad plug-in — it’s extremely useful and fun to play around with. The Nyquist language is a variation of Lisp, which we love for scripting. The built-in functions are interesting and lend themselves easily to higher-level functions. However, we are interested in expanding on the current implementation of Nyquist, which has been in Audacity since the inception of the program. We want to learn enough about sound design to create a reference page that will be just as legible to programmers as to sound editors. In addition, we hope to create a simple tutorial to introduce programmers to the concept of scripting with sound. Our biggest goal is to wrap the current Nyquist implementation in our MediaScript interface to encourage scripting and make it a simpler and more interactive experience.

Frustrations with PyGIMP

I’d like to complain for a few seconds if you’re up to listening to my rants. In my exploration of the Programming Language Python, especially how it is already implemented in GIMP, I’ve come across one of the most irritating “features” I’ve had to deal with as a computer scientist. Currently there are two ways a programmer can interact with images by writing code in Python. First, included in every copy of GIMP is “Python-Fu,” an interactive scripting window. It works in nearly the same way as the Script-Fu extension, except it supports Python rather than Scheme. This means it opens up an interactions pane where programmers can declare functions one line at a time or call them by loading a “.py” file that contains their pre-written script. This doesn’t exactly scream interactivity, so for all practical purposes it completely fails to meet our goal for interactive scripting. In addition, programmers have the ability to write plug-ins for GIMP using the “PyGIMP” Package. These plug-ins must be written before GIMP is even opened, which also rules them out for creating an interactive scripting environment.

So this sounds pretty cool right?  How is this frustrating you ask?  Well, imagine trying to write a book in French with only a Latin dictionary. Well, that’s kind of like trying to script in PyGIMP. GIMP offers the GIMP Procedural DataBase (PDB), which, like the name suggests, is a database of all the internal procedures for GIMP. To write a script in Python, a programmer has to find the procedure they wish to call, change all the dashes to underscores and hope it’s an actual PyGIMP procedure. For example, in the GIMP PDB, to access the compositing color of a channel you use “gimp-channel-get-color,” but when I try and call “gimp_channel_get_color” in Python-Fu, which should do the same thing, I get an error saying the procedure doesn’t exist. Aside from the fact that programmers have to change dashes to underscores, which is reasonable since Python interprets dashes not surrounded by spaces as a minus sign, why is there a built in procedure that doesn’t exist in PyGIMP but does in GIMP? Even more, why does no one seem to have an actual list of Python Procedures for GIMP? This wouldn’t be necessary if there was a one-to-one working correlation from the PDB to PyGIMP, but there isn’t. Ugh. I guess this means I have one more step if I want to create a good solid Python Implementation.

Making the switch to color

One of my favorite aspects of scripting GIMP is the flexibility it offers when I’m playing with image models and want to make one or two changes and then see the difference.  Case in point: modeling the ‘recgons’ that I’ve been working on.  These past few days, I’ve been experimenting with putting a little color into the recgons.  At first, I thought I would just add a call to the (random) procedure and use a randomly selected color for each recursion, but I quickly realized that this didn’t offer the level of control that I was after.  Instead, I removed the recursion level parameter from my recgon procedure and replaced it with a color list parameter.  The user can now control the color of each level of recursion through the order of his or her color list, and recursion depth is dependent upon the length of the color list.  This allows creation of cool images like this one:

A seven-sided recgon from a list of four blues and two whites.

At this point, I realized that I would need somewhere to house examples of the images that I’d created, so that they could be easily shared.  This led to my big side project: learning HTML such that I could build a website.  The basics were easy enough – wrapping text in tags like <h1> </h1> was a concept that I understood from various blogging experiences, and other ideas seemed fairly intuitive.  Now I’m working on more complex concepts like style and more intricate formatting.  You can view the website here.

At the same time, I’m continuing to work my way through Visual Modeling with Logo, expanding upon some of the other models.  My project for the next day or two is finding a way to expand and generalize the (image-draw-nephroid!) procedure that we wrote.

A nephroid

Here’s an example of a nephroid.  Currently, it’s drawn by repeating circles of varying radii.  However, I’d like to further generalize it so that it can draw by repeating any shape – stars, rectangles, etc.  I’d also like to add a bit of color – but since the nephroid doesn’t have a recursion level parameter but rather a frequency parameter that tends to be a high number, I need to use a different strategy than with my recgons.

Self-Disclosing Design Tools

We need to encourage more people to use interactive application scripting in addition to the GUI. Users with little programming experience might be intimidated by using or writing scripts. One way to help familiarize users with scripting is through self-disclosing tools. We will display scripting commands that correspond to graphical manipulation. These actions, represented as scripts, will show up in a log window, so that users of the GUI can incidentally learn syntax through observation and become more comfortable with scripting in an application. For example, if a user changes to foreground color to red using their mouse in GIMP, the self-disclosing log would show: (context-set-fgcolor! “red”).

All the previous actions will be recorded in the log for reference and repeatability. These scripts provide a gateway for the typical GUI user into interactive scripting: they can copy the code from the log and then modify and run these scripts in the interactions pane, thus learning the syntax and functionality of the scripting language in the application. Self-disclosing tools can encourage users to transition from a fully graphical interface to incorporating a script-based programming environment in their regular application use.

We will focus on implementing a logging interface for GIMP that displays Scheme script commands to encourage more users to script with MediaScript. Changes of settings, such as the brush type, can be easily modeled with scripts such as (context-set-brush! “Circle Fuzzy (07)”). However, it is important to consider how more complex actions would be modeled, such as the creation of a non-linear line using the paintbrush tool. A single graphical action can be modeled in multiple ways and many design decisions must be made. A non-linear line could be modeled with multiple or iterative (image-blot! image col row) commands or with multiple calls to (image-draw-line! image c1 r1 c2 r2) to simulate the drawing by connecting lines from point-to-point.

There are many other design obstacles we face in creation of this tool- some actions cannot be easily modeled with scripting, such as changing preferences, copying, or saving. We also must construct a consistent and detailed practice from representing actions, which can often be modeled with scripting in a variety of ways. Throughout this project, we will be considering this and many more design and implementation problems.

Currently, I am exploring how actions are represented internally in GIMP, so that we can record these actions in the scripting log. I have looked at GIMP’s undo stack which stores many past actions, and I plan on tracing an action through the GIMP source code to see how it is represented in GIMP.

Applying Scripting to a New Domain

Using the MediaScript program implemented by Dr. Rebelsky and past undergraduate research teams, budding coders and designers have been scripting in the Gnu Image Manipulation Program (GIMP) for years. We’re well aware of the benefits of scripting in an image-editing program (as described by Dr. Rebelsky in our first post). With the success of the GIMP plug-in in mind, we’ve been looking for other useful applications of interactive scripting. My current goal for the summer is to develop a MediaScript plug-in for Audacity, an open-source program for audio editing.

Before we started working with Audacity, we wanted to make sure that scripting would be useful in the audio domain. After interviewing an experienced theatrical sound designer and researching, we found that many problems encountered in current sound design programs could indeed be solved by scripting. Precision is difficult to manage when one is working with a complicated stereo sound and a mouse, but it could be improved with the addition of a scripting interface. Collaboration and post-production customization of sounds is much easier when a script is available to generate or edit a sound (as opposed to the transfer of huge .mp3 or .flac files, neither of which are easily edited except in rudimentary ways). Randomness is also an interesting application for scripting; if a background track is randomized, one may avoid noticeable and annoying loops in a sound. Scripting could definitely make life easier for a sound designer, and would allow for new creative possibilities for both designers and programmers who have never worked with sound before.

As I started looking through the current plug-ins, I realized that scripting for Audacity was just as useful as we’d hoped. In fact, it was so useful that a scripting plug-in called Nyquist already exists. With the plug-in, one may interactively script or create new plug-ins to be displayed in Audacity’s menu. The plug-in also allows the creation of more plug-ins in the Nyquist language (a dialect of Lisp which is quite similar to our current scripting language, Scheme). Upon discovering this widely-used Audacity plug-in, we decided to take our program in a new direction.

I plan to learn Nyquist well enough to write programs that will fulfill our original hopes for our MediaScript plug-in. Once I am familiar enough with the language, we hope to modify Nyquist to make it more scripting-friendly and to include higher-level functions for greater usability. If the modifications we have made by the end of the summer are useful and complete enough, we plan to release them as an open-source plug-in for Audacity. Hopefully, an interface that better stresses the benefits of interactive scripting will help more people understand the power of this paradigm.

Programming Language Applications

So I know what you’re thinking. Why Python and Scheme? Well, originally, Dr. Rebelsky chose Scheme because Grinnell uses it for the introductory Computer Science classes and because of its elegance, beauty, and support for higher-order programming.  However, Scheme is not a particularly popular programming language.  In fact, one of the reasons Scheme is used as the introductory language is to try and ensure everyone entering the class knows just about the same amount of syntax.

To better promote the notion of interactive application scripting, and to understand how IAS relates to the language domain, we will explore the development of an IAS interface for Python in the GIMP.  The way we see it, learning the language is one of the biggest barriers to entry for IAS. Since Python is one of the two most popular scripting languages, we hope that by adding a Python interpreter we can vastly increase the IAS potential user pool.  In addition, Python’s simple structure and style make it not only very easy to learn, but also fun to work with, which means new users will find it much easier to start scripting interactively.  Python is also multi-paradigm — providing deeper support for object-oriented programming than does Scheme, thus allowing future research teams and designers to explore the applicability of the object-oriented model to interactive application scripting.

Currently, our version of the GIMP allows basic scripting in both Python and Scheme with the standard Python extension and the Scheme interpreter written by Dr. Rebelsky and his past research students.  While the MediaScript extension has added a vast amount of functionality for Scheme, the current Python add-on is not nearly as well adapted.  Therefore, we plan on adding Python to the current MediaScript extension and extending Python’s application library to further expand its user base.