Interactive Ambient and Tangible Devices for Knowledge Mobilization

 InfoInfo
Search:    

Read more about this research project on [WWW]Digital History Hacks
Go to the Lab for Humanistic Fabrication

Summary of Proposed Research

One of the key problems that social scientists and humanists face is knowledge mobilization: getting information out of the ‘silos’ surrounding particular research groups, integrating it on a broad scale, and making it available to all Canadians. The transformation of image, text and sound into a common digital currency has profoundly lowered the transaction costs for researchers to find and utilize new information. A range of new technologies—powerful search engines, wikis, weblogs, text and data mining tools, and so on—make it easier and faster than ever to conduct research and disseminate results. In many disciplines, however, the focus has remained on individuals reading and writing with traditional desktop or laptop computers. Recognizing that there is potential for integration and dissemination of knowledge, SSHRC created a program to support these activities for strategic knowledge clusters. (This program does not directly fund research activities, however.) The successful development of these clusters, in turn, raises new research questions about the best ways to mobilize knowledge within and beyond clusters.

I propose to develop a methodology and a number of prototype devices to make the digital data sets and interpretations of a strategic knowledge cluster available in interactive, ambient and tangible forms that can be recreated in many different settings. To give some idea of the potential of these kinds of devices, consider the difference between writing with a word processor and stepping on the brake of a moving car. While using a word processor you are typically focused on the task and aware that you are interacting with a computer. The interface is intricate, sensorimotor involvement is mostly limited to looking and typing, and your surrounding environment recedes into the background of awareness. On the other hand, when braking you are focused on your involvement with the environment. Sensorimotor experiences are immersive, the interface to the car is as simple as possible, and you are not aware that you are interacting with computers (although recent-model cars in fact have dozens of continuously operating microcontrollers). As academic researchers we have tended to emphasize opportunities for dissemination that require our audience to be passive, focused and isolated from one another and from their surroundings. We need to supplement that model by building some of our research findings into communicative devices that are transparently easy to use, provide ambient feedback, and are closely coupled with the surrounding environment.

I propose to create prototypes of such devices for a community of researchers and stakeholders known as [WWW]NiCHE: Network in Canadian History & Environment. NiCHE is a strategic knowledge cluster that has SSHRC funding through 2014. This project will give NiCHE members early exposure to recent developments in pervasive and physical computing, interaction design and desktop fabrication. In keeping with NiCHE’s policy of open source and open access, all plans, software, and tutorial materials from the project will be released as open source. This project will give NiCHE new ways to communicate with the public, will facilitate new partnerships by building links with artist-technologists working in the open source hardware community, and will help to foster the emergence of a network of humanist “makers.”

Detailed Description of Proposed Research

The development of knowledge clusters in the Social Sciences and Humanities has created a need for new research on knowledge mobilization. What are the best ways to integrate research findings and present them to as wide an audience as possible? Here I propose to develop a methodology and a number of prototype devices that will communicate the research results of a knowledge cluster in interactive, ambient and tangible forms. To supplement more traditional modes of communication, these devices are designed to provide information at the periphery of awareness in natural settings. They are meant to be transparently easy to use. Instructions for building and using these devices will be released as open source hardware (Torrone 2007), making it easy for others to re-implement them and adapt them to new communicative contexts. This project builds on recent developments in interaction design, pervasive computing and desktop fabrication, subjects that may be unfamiliar to many researchers in the Social Sciences and Humanities. It also taps into the recent enthusiasm for ‘making’ (Parks 2005) and the emergence of a community of people who are interested in developing high-tech, do-it-yourself open source projects.

I propose to build four kinds of device. Each makes use of some combination of digitized images, texts and sounds to communicate research results in ambient or tangible form. Each has a software component that can be widely used without any specialized hardware, and a physical component that can be recreated by others and modified for a variety of settings. Taken together, the devices demonstrate the potential of a new approach to knowledge mobilization, foreground some of the questions raised by recent technological development and provide a foundation for further research.

Dashboard. This is a wall-mounted display in a busy corridor that provides a constantly updated overview of the research activities of NiCHE, a SSHRC-funded knowledge cluster. It is a straightforward example of what interaction designers call an information appliance (Norman 1999), a networked device that provides specific, up-to-the-minute information in a natural setting. From a distance, the dashboard displays a computer-generated imagescape (described below) to stimulate the curiosity of people passing by. As you approach, it switches to an overview of current NiCHE research activities—like the posters displayed outside many laboratories, but constantly updated since it is connected to a server. NiCHE members have the option of swiping a radiofrequency ID tab near the dashboard to see a user-customizable display showing personal information like an agenda, recent messages or whether particular collaborators are currently online. When the viewer steps away from the dashboard it reverts to the public display. The prototype will be constructed using infrared and ultrasonic proximity detectors, an RFID reader and a microcontroller board purchased with ITST funds, and an LCD display and computer provided by NiCHE. A mobile prototype will also be built to display at workshops, using a tablet computer purchased with a previous SSHRC RDI grant (Turkel 2005). The dashboard demonstrates how digital images and text can provide information at-a-glance, automatically customized for a variety of different kinds of stakeholders (Gelernter 1992). The three levels of information (ambient imagescape, general information for the public, and customized information for stakeholders) are also provided online so that they can be viewed with any computer connected to the internet. (Although in this case, of course, the information would not be ambient since the user would have to sit down at a computer and visit the NiCHE website.) Part of the instruction package developed for the dashboard will suggest ways that it might be modified for use in other settings. In a public history context like a museum exhibit, the RFID tabs might be used to customize displays to provide material taking the different points of view of various people from the past. An interactive exhibit about a nineteenth-century manor could look very different to people depending on whether they were holding ‘family’, ‘servant’, or ‘visitor’ tags, for example.

Imagescapes and soundscapes. NiCHE activities include the compilation and interpretation of large data sets (e.g., historical climate data). We are currently developing text and data mining tools to assist researchers in the study and understanding of these data. Psychological research suggests, however, that conscious perception makes use of only a fraction of the total bandwidth of sensory processing, that the majority of what we experience is actually subliminal (Norretranders 1999). We need to develop communicative tools that work at the periphery of awareness. As Mark Weiser, one of the pioneers of pervasive computing argued, information provided at this level can be part of a “calm technology,” something “we are attuned to without attending to explicitly” (Weiser & Brown 1995). Soundscapes and imagescapes are reconfigurable devices that present auralizations and visualizations of live, multidimensional data sets, giving researchers and stakeholders an augmented capacity to understand the research data. Both devices will be built on top of a software mining and summarization module which creates a continuously-updated stream of data. The prototype soundscape will use the data to parameterize a software audio synthesizer written in [WWW]Max/MSP or PD. Using the system, you might hear the sound of wind in the trees, the lapping of water, calls of various birds, each sound representing a time series or the occurrence of a particular event. Bird calls might be used to represent historical measurements of transnational species migration, water noises to represent rainfall or humidity, and so on. When a soundscape is playing in the background it will fade from conscious awareness, drawing attention only when the data change abruptly. You could easily reconfigure the system to monitor the online activity of NiCHE members instead. Then the gradually increasing volume of, say, bird songs would give you a sense of a distributed community coming together to work. The same data will also be used to parameterize [WWW]Terragen, a freely available artificial landscape generator. In this case, slowly changing landscape features provide a calm visualization that can be projected into a common area or used as a screensaver. Both imagescapes and soundscapes will be streamed over the Internet from the NiCHE server. The information kit for these devices will provide instructions for researchers who wish to input their own data or to realize it in different forms. In the context of a museum exhibit on social history, for example, demographic or census data might be used to parameterize the synthesis of barely audible voice-like sounds. In a different context, the streaming data might be used to control the behaviour of a colony of small robotic creatures in an artificial terrarium, adding an artificial life ‘visualization’ to the soundscape and imagescape (Hrynkiw & Tilden 2002; Wilson, 2002; Whitelaw 2006).

Geo-DJ. This project builds on my previous SSHRC-funded research on place-based computing (Turkel 2005). In the earlier project I developed GPS-enabled handheld and tablet computer systems to display archival sources that were targeted to the user’s location. While walking around a present-day environment, for example, they were able to see their location on a historical map, call up old photographs and newspaper articles about the place, and so on. Prototype place-based systems were demonstrated for NiCHE members and for Museum London in the summer of 2006. The geo-DJ will expand this work into an ambient, auditory dimension. Imagine walking around outside with an iPod-like device that is playing an electronic soundtrack. The music changes as you move, reflecting the historical land use patterns of the area that you are exploring. You may choose to represent patches of original forest with a flute, a dairy farm with bass viol and cow bells, a factory with a percussion ensemble, a slaughterhouse with discordant horns. As you walk towards the site of an old factory, the sounds of percussion rise in volume to dominate the music. The geo-DJ will use GPS to determine your present position, and a geographical information system (GIS) to calculate your distance and direction from the centroids of the historical features of interest. That data will then be used to mix the audio tracks that represent each feature. The initial prototype of the geo- DJ will be based on a GPS-enabled handheld computer purchased with the earlier grant. Subsequent versions will make use of progressively more specialized hardware. I anticipate that this device will be particularly popular among our new scholars, many of whom are almost as passionate about ‘headphone space’ as they are the natural environment. The information kit for the geo-DJ will show users how to prepare their own geodata for use in the system, and how to map features of interest to more explicit audio cues. For example, an industrial archaeologist looking for traces of a timber waggonway might configure the geo-DJ simply to chirp every time the present location matched the position of the track on a historical map. The device would then function more like a metal detector, albeit one that could be programmed to ‘detect’ any feature that appeared on a historical map.

Ice cores and tree rings. Paleoclimatologists use proxy evidence to reconstruct past climates. Core samples taken from layers of ice deposited in Greenland, for example, provide a record of the history of temperature and precipitation, airborne pollution, vegetation (from pollen), and so on. The artificial ice cores and tree rings that I propose to build will be tangible browsers that allow the user to explore digital models of climate history by manipulating physical interfaces that are based on this proxy evidence. The ice core will be a stratified column of plastic that is cold to the touch. By moving a finger along it, the user will be able to browse back and forth through a historical climate database compiled by NiCHE members. As with real ice core samples, deeper layers will correspond to earlier times, and layers nearer the top surface to more recent ones. The artificial ice core will be created on a CNC milling machine, will be cooled with a peltier junction, and will make use of capacitive touch sensors and infrared to detect finger position. The artificial tree rings will also be created by milling and will work in a similar manner, allowing the user to manipulate a physical model as an interface into a dendrochronological database. One of the arguments in favour of tangible computing is that our skills for physical interaction with material objects are highly developed, and that it makes a difference to our understanding if we are able to physically manipulate models rather than interacting with them at a remove (Ishii & Ullmer 1997). As with the other devices except the geo-DJ, the databases will be on the NiCHE server and accessible via the internet. The information kit for the ice cores and tree rings will include CAD/CAM files and schematics so that they can be recreated by anyone with the appropriate equipment. If there is interest, we can also make tangible interfaces like these available in prefabricated kit form to be used for teaching purposes.

These interactive, ambient and tangible devices will draw on the data, tools and methodologies that are currently being developed by an interdisciplinary group of digital humanists (Cohen & Rosenzweig 2005; Turkel 2005-present). They will also draw on recent technological developments in three related fields that are reshaping the sites where digital information is transformed into material forms.

Pervasive computing. Thanks to continual advances in microelectronics, we are at the point where computational devices are becoming ubiquitous, invisible, part of the surroundings (McCullough 2004). The design theorist Adam Greenfield refers to this condition as “everyware” (2006). A number of technologies work together to make this possible. Embedded microprocessors put the power of full computers into tiny packages. Micro-electro-mechanical systems (MEMS) include sensors and actuators to sense and control the environment. Radio transceivers allow these miniature devices to communicate with one another and get online. Passive radio frequency ID circuits (RFIDs) are powered by radio waves to transmit identifying information. All of these systems are mass-produced so that unit costs are very low, and it becomes possible to imagine practically everything being manufactured with its own unique identifier and web address. This scenario is sometimes called the “internet of things.” Someday instead of searching for your keys you may be able to Google for them instead. As Bruce Sterling notes, practically everything in the world could become the “protagonist of a documented process” (2005). Provenance has typically had to be reconstructed painstakingly for a tiny handful of objects. Most historians are not ready to conduct research in a world where every object can tell us about its own history of manufacture, ownership, use, repair, and so on. Dealing with pervasive computation will require the ability to quickly focus on essential information, to relegate non-essential information to peripheral awareness, and to access information in the places and settings where it can make a difference.

Interaction design. The insinuation of computation and interactivity into every conceivable setting has forced designers to abandon the traditional idea of “human-computer interaction,” and to take a much more expansive perspective instead (Moggridge 2006; Saffer 2006). Not only is everything becoming a potential interface, but many smart devices are better conceptualized as mediating between people, rather than between person and machine. Services like ordering a cup of coffee at Starbucks are now designed using the same techniques as those used to create interactive software (e.g., Google calendar) and hardware (e.g., the iPod). In order to benefit from the lessons of interaction design, historians will have to take into account the wide range of new settings where we can design experiences and shape historical consciousness. The technology of tangible computing provides a link between pervasive devices, social interaction, and the material environment (Dourish 2004).

Desktop fabrication. Most radical of all, everything that is in digital form can be materialized, via machines that add or subtract matter. The former include a range of 3D printing technologies that deposit tiny amounts of glue, plastic or other materials, or that use lasers to selectively fuse small particles of metal, ceramic or plastic. The latter include computer-controlled milling machines, lathes, drills, grinders, laser cutters and other tools. The cost of these devices has been dropping rapidly, while their ease-of-use increases. The physicist Neil Gershenfeld has assembled a number of “fab labs”—universal fabrication laboratories—from collections of these devices. At present, a complete fab lab costs around $30-$40,000 and a few key machines are considerably cheaper (Gershenfeld 2000, 2007). Enthusiasts talk about the possibility of downloading open source plans and “printing out” a bicycle, an electric guitar, anything really. An open source hardware community is blossoming, aided in part by O’Reilly Media’s popular new Make magazine, and drawing on the work of an earlier community of artist-technologists who developed physical computing (Wilson 2002; Igoe & O’Sullivan 2004). Desktop fabrication will make it possible to build and share interactive devices that communicate our knowledge in novel, material forms.

I will use some of the key ideas, techniques and goals of these three research areas to create interactive, ambient and tangible devices for members and stakeholders of NiCHE: Network in Canadian History & Environment. Founded in 2004, NiCHE is a SSHRC-funded strategic knowledge cluster with over a hundred participating members and more than thirty national and international partners, including government agencies, museums and archives, scholarly societies, environmental groups and First Nations. Although pervasive computing, interaction design and desktop fabrication are somewhat removed from the ambit of most of our members and partners, this project will make NiCHE researchers more familiar with the technology and will integrate it into their research practice.

Using ITST funds and in-kind contributions from NiCHE, I will hold two hands-on summer workshops in Washington DC (2008) and Toronto (2009) focused on designing, building and using these knowledge mobilization devices. The workshops will include demonstrations of existing systems and short courses in fabrication and programming. They will bring NiCHE members together with artist-technologists from the open source hardware community, academic delegates from Congress and local educators and public historians. In developing the workshops, I will also draw on my pre-existing collaboration with researchers at the [WWW]Center for History and New Media (CHNM) at George Mason University, one of NiCHE’s institutional partners. CHNM is the largest digital history group in North America (and possibly in the world), sponsoring more than two dozen online projects and providing free, open source research tools and other resources to digital humanists.

Timeline of Activities

References
This is a Wiki Spot wiki. Wiki Spot is a non-profit organization that helps communities collaborate via wikis.