EnCue by Octava link

IRC projects emerge and progress along a variety of pathways. Some are generated in-house by the IRC staff. Others grow from collaborations with faculty, who come to us with ideas and funding, or take part in our Summer Faculty Research Fellowship Program. Projects have different lifespans as well. Some end in a single film or website; others use the IRC as an incubator and go on to independent production, like EnCue.

Professor of Music Linda Dusman applied twice to the IRC Summer Faculty Fellowship program, and never received an award. But she doggedly found other funding sources, and over the course of several years brought her idea for an app to enhance orchestral and symphonic performances to reality. EnCue by Octava (the company that Dusman founded with her collaborator, Eric Smallwood) is now being used regularly by the London Symphony Orchestra, the National Orchestral Institute, and the Royal Philharmonic Orchestra.

EnCue allows orchestras to deliver commentary about music and musicians in real time on mobile devices. The information scrolls smoothly by, on a dark screen, in a way that enhances rather than distracts from the performance. The ultra-slow animations keep the focus on the music and the content brings wandering attention back to the details of the musical performance at special moments during a piece.

A sample screen image

Dusman had the idea for EnCue (then known as Symphony Interactive) in 2010, during a UMBC Orchestra concert, when she found herself whispering details about the music to UMBC President Freeman A. Hrabowski III. At the same time, she realized how the newly emerged smartphone and tablet computer allowed for new kinds of interactions with devices. She found some funding and brought her idea to the IRC. There, around the conference table, she was put together with Eric Smallwood, a UMBC graduate (BA and MFA) who was the IRC’s Technical Director for animation at the time. Dusman and Smallwood, along with other IRC staff and interns worked up version 2.0 which they tested at an orchestra rehearsal. The app crashed, but the team kept at it. Version 2.1 showed more promise but eventually crashed as well.

Around this point, Dusman and Smallwood began working with the Maryland Innovation Initiative, a program of TEDCO, an independent organization created by the Maryland Legislature "to assist in the creation and growth of technology-based businesses" in partnership with select research universities in the State They received mentoring as they wrote a business plan and grappled with patent and intellectual property issues. After three tries, they received funding and were able to hire a professional computer programmer. By this point, Smallwood was working as a faculty member in Visual Arts, and the IRC was no longer involved. This is what should happen as a project progresses from research phase to a production and business phase.

EnCue at a rehearsal of the London Symphony Orchestra

It took many more versions, rounds of funding, and a few years for the app to reach its current commercial version. Orchestras can draw on EnCue’s library of content (about thirty pieces), contract with the EnCue team for custom content, or write their own. One of the biggest challenges was working out server issues, so that multiple orchestras can use the software simultaneously.

Another sample image, showing the simple, unobtrusive graphics

What’s really exciting about this project is its potential for use in all sorts of ways, not just for orchestral performances. It could easily be adapted for theatrical or dance works, for example. But the key is that the software running the app is really a system that delivers content in real time on mobile devices. So it could be used at conferences, for example, to annotate presentations.

Regardless of where it goes next, EnCue is an IRC success story.

Anne Sarah Rubin

May 9, 2018

Experiencing Collaboration link

The first article about the Art of Transformation project is published.

There is increasing chatter about collaboration all around and among us at the university. To collaborate is becoming an aspiration. So many of us want to reach out and work together, preferably on something meaningful, and the process itself is meaningful. At a time when national leaders struggle to work together to meet the public's critical challenges, it feels good to seriously engage one another—particularly across the boundaries that so often isolate and divide us. In this spirit we want to share a newly published article about a project the IRC has been deeply involved in, The Art of Transformation: Cultural Organizing by Reinventing Media. It appears in the current issue of PUBLIC, the journal of the Imagining America consortium. It is about the process and practice of university people and communities working toward engaging with one another in a way that is truly collaborative.

If you have not been made aware of, or been able to join the many UMBC faculty, students, staff and administration that have become involved with Imagining America (IA), this article should give you a feel for what it's all about. IA is: "Publicly engaged artists, designers, scholars, and community activists working toward the democratic transformation of higher education and civic life." A lofty goal to be sure, but as we hope the article conveys, it is one that is surprisingly and wonderfully possible.

Writing the article itself was a deeply collaborative effort. Six of us wrote it together. Lots of articles have co-authors and many readers will be familiar with the process of writing and publishing an article or edited volume following standard protocols. There is a lead author who pulls together the thoughts of co-authors into an article that is clear and written with the single voice. It is a practical and successful tradition. Nonetheless, it was not applicable for writing about Art of Transformation because the very foundation of the project is about not privileging one voice over another. So with no one of us put "in-charge" we worked together, each adding what we could, until the writing was done. Our names are listed alphabetically rather than in an order that would describe a hierarchy of responsibility and involvement. We had no idea how to write the article when we began—people had very different ideas about structure and content—but getting the pieces to work together didn't feel like a series of compromises. It felt natural. Each voice found its place. Though writing a 9,000 word article is always a major untaking, in retrospect, this was probably no harder, and took no longer, than it normally is. We only had to rewrite the whole thing once!

Filming interviews on the street in Morrell Park

The article begins in a time beginning shortly after UMBC hosted the Imagining America conference in 2015. Conference organizers, which included UMBC and nearly fifty other organizations and institutions in Baltimore, pledged to stay organized—to keep meeting, at least long enough to launch some kind of continued collaboration. That collaboration became the Art of Transformation project. The article chronicles the first phase of the project, in which UMBC participants worked with four Baltimore communities and organizations including CultureWorks, WombWork Productions, and Chesapeake Center for the Arts, and with those involved in other UMBC projects such as Baltimore Traces, and Mill Stories. In each community, we listened to people and documented their stories about living in their communities—what they'd seen in their lifetimes, what it means to them, and what they imagine for the future. We captured and those stories in films and visualizations that will be organized and put into a new kind of public media during phase two of the project. The process of recording, editing, analyzing and compiling, then playing media back to those who have been recorded, definitely had ups, downs and bumps along the road. Given the often-troubled relationship between researchers and community residents, or as it's often phrased, "town and gown," pursuing the project in a way that upheld the spirit of collaboration was, like writing the article, challenging but possible and is paving the way for more possibilities going forward. We are encouraged, and together with our community partners, anxious to go much further along this path.

Both the article and the project it describes are things we at the IRC are proud to be involved with. The project is a great match for us because it combines reinventing media and rethinking and reworking the practices we use to create the stories we tell ourselves, about ourselves, and how we tell them. This is what IRC work is all about. Please check it out, and have a look around the rest of the issue once you do. In alphabetical order, the article's authors are: Frank Anderson, Doctoral Student, Language, Literacy and Culture and Assistant Director, Choice Program, UMBC; Beverly Bickel, Clinical Associate Professor, Language, Literacy and Culture, UMBC; Lee Boot, Director, Imaging Research Center, UMBC; Sherella Cupid, Doctoral Student, Language, Literacy and Culture, UMBC; Denise Griffin Johnson, Cultural Organizer, Culture Works & USDAC: Christopher Kojzar, Graduate Student, Intermedia and Digital Arts, UMBC.

Lee Boot

January 30, 2018

Faculty Fellows Work: Shabamanetica link

One of the most popular installations at the 2017 Baltimore Light City Festival was Eric Dyer’s Shabamanetica, two spinning sculptures that resembled massive ship’s wheels. As the wheels spin, a strobe light flashes, and the static images on the zoetropes come to life. One features spinning parasols and Panamanian waterfalls; the other shows Shanghai cargo cyclists and a machine that spits out poop-emoji pillows. Shabamanetica’s swirl of colors and objects unites three disparate places: SHAnghai, BAltimore, and the PanaMA Canal with Kinetics.. As Dyer explains, “The animations combine imagery from Shanghai, Panama, and Baltimore: three places connected anew by the recent expansion of the Panama Canal and dredging of the Port of Baltimore in preparation for the gigantic Neopanamax container ships.” This work was supported by one of the IRC’s Summer Faculty Research Fellowships.

Shabamanetica on display at Light City.

Dyer knew that he wanted the project to explore the connections between Baltimore, the East coast’s port that is best-suited to the enormous Neopanamax ships, and Shanghai, where Dyer had recently taught a zoetrope-making workshop. (Zoetropes are optical devices that give the illusion of motion from a series of static images.) While in Shanghai, he found himself fascinated by the wide variety of umbrellas and parasols that people used. Dyer speculates that he was drawn to them by their zoetrope-like radial shape and ease of spinning, and he shot video of his students playing with them. While visiting the Baltimore Museum of Industry, Dyer discovered that Baltimore was once the largest manufacturer of umbrellas in the world. But the Panama Canal, which made it possible to ship goods more cheaply from the Pacific Rim to the Atlantic, ended that.

Dyer drew on the BMI’s collections, including actual Baltimore umbrellas, and commercials for the “Teeny Popper.” As Dyer recalled, “the actress in the ad has this fantastic look of confusion and distress when she notices it’s starting to rain - to me she was wondering “what happened to our industries?’ There’s another great shot I used of her opening and closing the Teeny Popper, which I looped forward and backwards to make it appear as if she is opening and closing the umbrella endlessly.”

A close up shot showing the details of the umbrellas.

Dyer wanted Shabamanetica to create a seamless interactive experience for the public. He achieved it first by having individuals spin the wheels to activate the artwork. He also wanted to hide the technology, by putting the strobe light at the edge of the viewer’s peripheral vision and hiding the electronics in the base and behind the wheel.

This is where the IRC comes in. Our faculty research assistant, Mark Murnane, worked with Dyer to build the 400 watt LED strobe light system. The system uses an encoder to measure the speed at which the wheel is spinning and sync the strobe to it. The strobe light must be carefully synchronized with the wheel in order for the images to appear as a smooth animation. One challenge came in getting powerful-enough circuitry to drive 400 watts of LEDs. The other came in getting the strobe light to appear smooth at lower speeds, so as not to trigger seizures in any visitors.

The circuit board designed by Mark Murnane to control the LED lighting.

Finally, Dyer discovered that because the temporal resolution (frame rate) of the images was sometimes double that of film and video and the printed resolution of the images was about 16x that of HD video, the combination made for imagery that seemed hyper-real and richly dimensional.

Shabamanetica (2017) - Eric Dyer, Artist from Eric Dyer on Vimeo.

Anne Sarah Rubin

September 21, 2017

Faculty Fellows Work: Plantelligence link

Within a city like Baltimore, the landscape is generally considered the ‘background’ for human activity – a largely undifferentiated expanse of green, without much thought about the actual plants that fill the space. But UMBC Professor of Visual Arts Lynn Cazabon wants to move those plants to the foreground, and she is working with the IRC to explore the ways that plants respond to the environment around them. She also wants people to think about species that thrive in urban areas, and to that end is working with one common species, Conyza canadensis, better known as horseweed, a native annual. While most people think of horseweed as a nuisance, or weed, Cazabon is fascinated by its adaptability to the stresses of living in human-created landscapes.

Cazabon received a 2016 IRC Summer Faculty Fellowship for her project Plantelligence. As she explains, “Plantelligence emerged from my interest in how plants perceive and respond to events occurring in their surrounding environment, as a means to bring attention to how global warming impacts the way plants and in turn urban landscapes are currently evolving. Recent research in the field of plant neurobiology has led to inquiries into the evolutionary purposes of the many ways that plants sense their surroundings, including through analogues to sight, hearing, touch, taste and smell as well as through perception of electrical, magnetic, and chemical input.” But, plants move and react at speeds below human perception—and this is where the IRC’s research in photogrammetry, 3D modeling, animation, and virtual reality comes into play. As Cazabon explained, “I am using time-lapse photography and photogrammetry to study the movements of growing plants in order to translate these movements for human perception through animation…My goal in using VR is to create an immersive environment for the viewer which blurs conventional distinctions between inside and outside.”

Plantelligence first took shape in the photogrammetry rig, where we took scans of growing plants every 30 minutes for about two months, ultimately generating 8 terabytes of data. But the plan to create a 3D time-lapse film ran into a few temporal snags. The first was that the plants did not grow as quickly as we would have hoped. The second issue involved the time needed for the scans themselves: it took the computer 3-4 days to process a model from each scan. It would have taken far too much time to process every single scan. As a result, faculty research assistant Mark Murnane is working on a way to process the images on UMBC’s High Performance Computing Facility, which will speed up the process enough to make it feasible in the future.

We also learned a lot about the challenges of scanning a plant, because the models that were initially generated needed a lot of cleaning up by hand. Technical director Ryan Zuber calls these irregular models, with holes and deformation, “crunchy,” and he went to work smoothing them out. He cleaned up one model plant, imposing quadrangular polygons on its surface, which allow the model to be textured and animated.

But, as Zuber and Cazabon realized, it’s not easy to create and animate a realistic plant that is designed to be seen individually and up close. The horseweed has many leaves, tiny hairs, and variable textures, all of which need to be able to move independently of each other, and all of which need to be seen at multiple stages of growth. Zuber is treating each leaf as an individual ‘character’ and has built a rig that can work for all of the leaves, regardless of their specific geometry. He studied time-lapse films of plants growing in order to get a sense of the way the leaves grow and unfurl, and is now able to animate the plant.

The next step involves placing the plant in VR space where people can interact with it, which Cazabon envisions as a generic, unadorned gallery space. The goal here is to bring the outside to the inside: to isolate the plant against a neutral space that is more ideal for human perception. The final step will be to bring the animated plant and gallery environment together with custom software that will enable a viewer to explore and interactively affect the plant’s growth.

Anne Sarah Rubin

July 14, 2017

Pinhole Photogrammetry link

Have you ever used a pinhole camera before? Maybe you made one to look at a solar eclipse, or as a summer camp project, or in art class. The pinhole camera, or camera obscura is one of the oldest image-making technologies we have, and here at the IRC we are seeing its potential to improve our photogrammetry work.

The principles behind the pinhole camera go all the way back to Euclid in 300 BCE, but Leonardo Da Vinci is widely credited with refining and improving the camera obscura. A camera obscura is a light-proof box (it could actually be an entire room) with one very small hole. The hole lets in light, and directly opposite the hole is an upside down projection of whatever is outside. This happens because light travels in a straight line, but is distorted or refracted as it passes through the hole. Da Vinci realized that this is essentially the same way that the human eye works-- our brain flips the image to right-side up. The smaller the hole, the sharper the image.

First published picture of camera obscura in Gemma Frisius' 1545 book De Radio Astronomica et Geometrica

What does this have to do with out 94-camera photogrammetry rig? Mark Murnane, the IRC's post-baccalaureate faculty research assistant is our photogrammetry expert, and over the course of his research has grappled with two related issues. The first is that when the photon of light hits any given camera, the computer has to figure out the location in the photo of the corresponding pixel. That location is a function of the angle from which the light is entering the camera. But, camera lenses actually complicate this process, because they add distortion. Lens distortion warps the image in a variety of ways, causing an image of a single point to appear as a larger blob. This blob is called a point spread function, and describes how photons entering from a particular angle move through the lense. In the sample image below, each star is a point spread function of a sharp point of light. In an ideal world, each photon would only light up one pixel. With practical pinhole lenses however, refraction causes the image of a single point to spread into an Airy disk.

Measured PSF of a traditional lens

Traditional lenses exhibit radial distortion, tangential distortion, skew, and other optical effects. Accounting for all of these can lead to a model with 16 or more parameters. With 94 cameras to solve together this leads to an incredibly difficult problem.

Measured angular changes in PSF of a pinhole lens
Measured angular changes in PSF of a traditional lens

But, a pinhole lens, especially one with an extremely small and precise hole (the one Mark tested is .2 mm) has much less distortion, and that distortion is also more predictable or regular. Furthermore, with the pinhole, the distortion doesn't vary by location within the image. The sample image from the pinhole (below) shows straighter lines, and therefore the math needed to correct it is correspondingly simpler, resulting in 3-parameter model. Having fewer free parameters means we can find a more accurate reconstruction in less time, and moving from 16 parameters to 3 makes a world of difference.

PSF of an ideal pinhole
PSF of a practical pinhole

Having a predictable point spread function also opens the door to programmatic image sharpening. By deconvolving the point spread function across a captured image, some detail lost in the capturing process may be recovered. This method has become a crucial tool in other fields such as astronomy, where it has been used to sharpen images taken from many observatories.

The trick is to figure out how to get a perfect pinhole, without any aberrations. The less precise the pinhole, the less detail in the image. However, once we have figured out the optimal pinhole, then we will switch all of the cameras in the photogrammetry rig to pinholes, which should in turn allow for more accurate scanning.

There is a trade-off being made with this: is less light on the individual camera sensors worth knowing exactly where the light came from? We think that it is.

Mark Murnane, Anne Sarah Rubin

June 14, 2017