Synthesis

( One of the principal scientists of psychoacoustics comments on the importance of a researcher being guided by his/her own felt experience in the study of subjective experience. )

At this point, I want to interject a few words about subjectivity and objectivity in psychological research. The personal experience of the researcher has not fared well as acceptable data for scientific psychology. Since the failure of Titchener’s Introspectionism, a very biased form of report of one’s experience, in the early twentieth century, and the rise of Behaviourism to replace it, scientific psychology has harboured a deep suspicion of the experience of the researcher as an acceptable tool in research.
You would think that the study of perception would be exempt from this suspicion, since the subject matter of the psychology of perception is supposed to be about how a person’s experience is derived from sensory input. Instead, academic psychology, in its behaviouristic zeal, redefined perception as the ability to respond differently to different stimuli – bringing it into the behaviourist framework. We may be doing research nowadays on cognitive processes, but the research methods are, on the whole, still restricted to behaviouristic ones. Since it was a perceptual experience of my own (the rapid sequence of unrelated sounds) that set me off on a 40-year period of study. of perceptual organization, I have always questioned the wisdom of this restriction.

In my many years of research on how and when a mixture of sounds will blend or be heard as separate sounds, my own personal experience and those of my students has played a central role in deciding what to study and how to study it. When I encouraged students to spend a lot of time listening to the stimuli and trying out different patterns of sound to see which ones would show the effect we were interested in, far into the academic year, and nearing the time that they should have been carrying out their experiments, they would get nervous and ask when they would start doing the “real research”. I told them that what they were doing now was the real research, and the formal experiment with subjects and statistics was just to convince other people.

Furthermore the role of subjectivity has often been criticized by journal reviewers: In the reviews of my first published article on auditory stream segregation, which showed that a rapid alternation of high and low sounds segregated into two perceptual streams, one of the skeptical reviewers proposed that there was something wrong with my loudspeakers – perhaps they continued to give out sound after the tone went off – and insisted that I test them.

I was convinced that if the reviewers had merely listened to the sounds, their objections would have evaporated, but in those days you didn't send in audio examples with your manuscript, and I’m not sure it would be acceptable for most journal editors even today.
…
Anyway, I got around the taboos about subjective data by giving many talks accompanied by auditory examples and by eventually publishing my own Compact Disk of auditory demonstrations. However, the CD didn't come until 23 years after the first research paper. Nowadays you could put demonstrations on the web and refer reviewers to the website.

Another thing that reviewers have criticized was the use of a subjective rating scale, asking listeners, for example, to rate on a 1 to 7 scale how clearly they could hear a sound in a mixture. Perception journals on the whole prefer tasks that involve accuracy. This is in keeping with the behaviouristic view of perception as the ability to make different responses to different stimuli. According to this view, you should be able to score the answers of the subjects as either correct or incorrect (For example by asking whether a particular sound was or was not present in a mixture of sounds) rather than simply accepting the listeners’ answers when they rate the clarity with which a target sound can be heard.

Sometimes we have used both types of measures, subjective rating scales and measures of accuracy, either in the same experiment or in a pair of related experiments. The two measures have given similar results, but the subjective rating scales have been more sensitive. I think the reason for their superiority is that they are a more direct measure of the experience, whereas turning one’s experience into the ability to form a discrimination between sounds brings in many other psychological processes that are involved in comparison and decision making.

As a result of my belief in experience as an important part of Psychology, I’m going to try to describe some of my research on auditory perception, but I won't give any data. Instead, I’m going to support my arguments with audio demonstrations to the extent that time permits. 

( One of the principal scientists of psychoacoustics comments on the importance of a researcher being guided by his/her own felt experience in the study of subjective experience. )

At this point, I want to interject a few words about subjectivity and objectivity in psychological research. The personal experience of the researcher has not fared well as acceptable data for scientific psychology. Since the failure of Titchener’s Introspectionism, a very biased form of report of one’s experience, in the early twentieth century, and the rise of Behaviourism to replace it, scientific psychology has harboured a deep suspicion of the experience of the researcher as an acceptable tool in research.
You would think that the study of perception would be exempt from this suspicion, since the subject matter of the psychology of perception is supposed to be about how a person’s experience is derived from sensory input. Instead, academic psychology, in its behaviouristic zeal, redefined perception as the ability to respond differently to different stimuli – bringing it into the behaviourist framework. We may be doing research nowadays on cognitive processes, but the research methods are, on the whole, still restricted to behaviouristic ones. Since it was a perceptual experience of my own (the rapid sequence of unrelated sounds) that set me off on a 40-year period of study. of perceptual organization, I have always questioned the wisdom of this restriction.

In my many years of research on how and when a mixture of sounds will blend or be heard as separate sounds, my own personal experience and those of my students has played a central role in deciding what to study and how to study it. When I encouraged students to spend a lot of time listening to the stimuli and trying out different patterns of sound to see which ones would show the effect we were interested in, far into the academic year, and nearing the time that they should have been carrying out their experiments, they would get nervous and ask when they would start doing the “real research”. I told them that what they were doing now was the real research, and the formal experiment with subjects and statistics was just to convince other people.

Furthermore the role of subjectivity has often been criticized by journal reviewers: In the reviews of my first published article on auditory stream segregation, which showed that a rapid alternation of high and low sounds segregated into two perceptual streams, one of the skeptical reviewers proposed that there was something wrong with my loudspeakers – perhaps they continued to give out sound after the tone went off – and insisted that I test them.

I was convinced that if the reviewers had merely listened to the sounds, their objections would have evaporated, but in those days you didn't send in audio examples with your manuscript, and I’m not sure it would be acceptable for most journal editors even today.
…
Anyway, I got around the taboos about subjective data by giving many talks accompanied by auditory examples and by eventually publishing my own Compact Disk of auditory demonstrations. However, the CD didn't come until 23 years after the first research paper. Nowadays you could put demonstrations on the web and refer reviewers to the website.

Another thing that reviewers have criticized was the use of a subjective rating scale, asking listeners, for example, to rate on a 1 to 7 scale how clearly they could hear a sound in a mixture. Perception journals on the whole prefer tasks that involve accuracy. This is in keeping with the behaviouristic view of perception as the ability to make different responses to different stimuli. According to this view, you should be able to score the answers of the subjects as either correct or incorrect (For example by asking whether a particular sound was or was not present in a mixture of sounds) rather than simply accepting the listeners’ answers when they rate the clarity with which a target sound can be heard.

Sometimes we have used both types of measures, subjective rating scales and measures of accuracy, either in the same experiment or in a pair of related experiments. The two measures have given similar results, but the subjective rating scales have been more sensitive. I think the reason for their superiority is that they are a more direct measure of the experience, whereas turning one’s experience into the ability to form a discrimination between sounds brings in many other psychological processes that are involved in comparison and decision making.

As a result of my belief in experience as an important part of Psychology, I’m going to try to describe some of my research on auditory perception, but I won't give any data. Instead, I’m going to support my arguments with audio demonstrations to the extent that time permits. 

Mirages & miracles / Exposition / Création décembre 2017, Les Subsistances, Lyon, France.
Une série d’installations qui abritent un animisme numérique.
Les œuvres, du petit au grand format, offrent toutes une coïncidence finement organisée entre virtuel et matériel : dessins augmentés, dispositifs d’illusions holographiques, casques de réalité virtuelle, projections grande échelle. Elles donnent à vivre un ensemble de scénarios improbables qui tiennent à la fois du mirage et du miracle, qui jouent à la frontière entre le vrai et le faux, l’animé et l’inanimé, l’authentique et l’imposture, la magie, le merveilleux, et l’inouï.

9:00 AM – 10:30 AM
3G “Game Studies 1: Roundtable with Ian Bogost” 
Ian Bogost (GaTech), Alenda Chang (UCSB, Edmond Y. Chang (Ohio), Heidi Coleman (Chicago), Patrick Jagoda (Chicago), Patrick LeMieux (UC Davis), Timothy Welsh (Loyola)

11:00 AM - 12:30 PM
Synthesis Open Atelier: Serra Vegetal Life and Other Scenarios  
Oana Suteu, Todd Ingalls, Sha Xin Wei, Brandon Mechtley, Chris Ziegler + Synthesis

Connor and others who are considering using machine learning.

(1)

Rebecca Fiebrink, a computer scientist @ Princeton now Goldsmiths, has created a very popular machine learning engine designed specifically for media artists: Wekinator - which can be used as a blackbox from Max or any client via OSC.   I think Lauren Hayes and several AME students have tried it out.  (Tell us what you think!)

http://www.wekinator.org/

Treat Wekinator’s machine learning engine like a power tool — it’s very useful for what it's designed for, but it can hurt you ( in mind and soul if not body ;) if you try to use it against its design.   Take appropriate courses from Pavan (or Suren, Robert…)  to learn when and why to use these tools.

Wekinator is not a research tool for those interested in inventing new methods in machine perception or signal processing, but Fiebrink has taken a great deal of care to user test, design and implement the best of known methods to 

 

            • Create mappings between gesture and computer sounds.

            • Creation of gesturally-controlled animations and games

            • Control interactive visual environments created in Processing, OpenFrameworks, or Quartz Composer, or game engines like Unity, using gestures sensed from webcam, Kinect, Arduino, etc.

            • Creation of systems for gesture analysis and feedback

            • Build classifiers to detect which gesture a user is performing. Use the identified gesture to control the computer or to inform the user how he’s doing.

            • Detect instrument, genre, pitch, rhythm, etc. of audio coming into the mic, and use this to control computer audio, visuals, etc.

            • Creation of other interactive systems in which the computer responds in real-time to some action performed by a human user (or users)

                        • Anything that can output OSC can be used as a controller

                        • Anything that can be controlled by OSC can be controlled by Wekinator

 

(2)

Having said that, these methods are also a trap in a fundamental sense: they can only recognize / categorize the given, and cannot produce novel, artful, living gesture.  (The non-prestatabilty of potential is the central thrust of the ontogenetics group.)  That’s why we make human-in-the-loop systems, technical ensembles.  

(3)

On the third hand, it could be quite fun to implement two ETUDES leveraging Wekinator’s power:

            Sha’s Follower-to-Leader game: Follow-spot tries at some point to anticipate and lead the human walker)

            Rawls’ N-1 game : Train on n walkers, remove one person, direct lightspot to move with the remaining n-1 walkers.

I’m pretty sure the Follower-to-Leader game would do something interesting.  I don’t see how to constrain the (repetition of the) walking to make the n-1 game work.

In 1995 I argued at Stanford & Berkeley that the master Wicked Problem in computer science is NOT NP≠P, but the brittleness of code. 

Instead of arguing with over-confident and under-informed colleagues during the dotcom boom, I started with some courageous, playful souls the Topological Media Lab* to create realtime, gestural media, and hybrid matter that can evolve with the robustness of analog materials, but with continuous (indefinite-dimensional) & multivalent state behavior that can be designed to suit the phenomena of interest, and admit ad hoc intention.   

To do it right is not trivial, requires a head for physics as well as a nose for irreality and poetry…

It’s time to reap…

* Georgia Tech / GVU / 2001-2004, Concordia University 2005-2013

Attached is a useful survey of how to design computational media things : tools, apps, services, even events, and performances with an aesthetic intent.

Some points / caveats:

• No one article can adequately represent all the different techniques, but this one serves as a good index.  (I'm sure Stacey K could point to a lot more :)

• This addresses the practical matter of how to design a sociotechnical thing: software application like a mobile app, a service like an augmented reality platform for use by curators or academic game developers, or an immersive responsive media environment for exploring Mars or terrestrial ecosystem, or an immersive environment for modeling buildings and streets, according to criteria that can be stated in advance:  for such and such a demographic, for a specific purpose -- like helping ASU seniors create scenarios about climate change for high school students.

• Such digital design methods are a practical and technical matter.  Research in digital design is NOT the same as using digital design to do research (which could be any inquiry to expand human knowledge -- eg why are we here?   Why are we mortal?  What is a gesture?   When should we replace some human work by machine work?   Etc etc etc)

• Digital design method is also NOT art.

 (Despite the authors' citation of Lev Manovich. )

• Experimental studies of experience, and (non-anthropocentric) experiential experiments, and in general any scientific (vs engineering) research  are NOT well served by pre-packaged methods.  I can talk about that.

Having said all that, the design methods surveyed are extremely useful when the sociotechnical context is stable, and it's time to make something work robustly enough to be used and useful.   Please study this carefully if you aim to making something "usable" by "other people."

The most valuable aspect of this article is its concrete observations about accommodating the putative "users" of (engineering lingo) / "audience" for (art lingo)  what you make in the very process of inventing and designing what you make.

Neither engineers nor artists are very good about this.  

That's why the Topological Media Lab was set up with a different ethical stance : NEITHER as an isolated engineering lab, NOR as an isolate artist but as home for fundamental inquiry that uses art and engineering research techniques.   So, can we do engineering or art research reflexively  (the makers are their own subjects and objects of study as well as the people they purport to help) and abductively?

Xin Wei