Re: Hadoop, pig, dynamical systems tools

Hi Brenda,

David Donoho has an extensive critical review of big data techniques in
http://courses.csail.mit.edu/18.337/2015/docs/50YearsDataScience.pdf

But I'd like to learn about HADOOP from your direct encounter, and understand how you think we can use such big data mining techniques to multi-dimensional activity / ensemble-body / gesture tracking.  It may be good to talk with Qiao to get his opinion on things.  I trust his scientific judgment most ...

And here's a curnudgeonly rant:

Glad to see you working thru all this,
Xin Wei

On Aug 8, 2016, at 8:40 PM, Brenda McCaffrey <brendamc@asu.edu> wrote:

Hi Xin Wei and friends,

Quick update:

*  Thanks to Connor for helping me get much of the Java script working within Eclipse.
*  I'm meeting with Mike K. tomorrow morning at 10am to look at the code.
*  I'm learning more about Hadoop and Apache Pig and it may be very exciting for us, although I have yet to find anyone in our realm who has knowledge of it.  Here are a few things I've learned:
  • Hadoop is an open source data management system that evolved from Google's behavioral search system about 10 years ago.
  • Hadoop is used by Amazon, Yahoo, Facebook and others to track and manage behavioral data.  For example, that's how Facebook knows where you move your mouse.
  • Hadoop allows massively parallel processing of large data sets.  It can be set up on unlimited low-cost computers to run infinitely parallel.
  • Apache Pig is a scripting language that manages Hadoop queries.
Apparently, Hadoop and Pig are primarily used in business systems.  The gentleman who wrote these dynamical systems codes (Jacob Perkins when he was at UT Austin), did so to demonstrate that one could solve complex scientific problems without massive data sets by using parallel processing.  Eureka!

I continue to be intrigued by this approach since it implies to me that we may be able to solve very complex, real-time dynamical systems problems by using relatively simple parallel computing frameworks in conjunction with Java.

I'm going to get the fundamental Java to work (I hope) to demonstrate it's functionality, but the real power of these programs is the ability to slice the simulations into partitions that can be evaluated in parallel to converge on the regions of interest.  This could be especially useful for time-domain data coming in from complex systems for which we don't know the closed form analytical models.  (Video?)

These are my thoughts for now.  I'm not a CS person so I may be completely in left field about this, but it looks promising.

Thanks for listening.
-Brenda


On Sun, Aug 7, 2016 at 4:10 AM, <shaxinwei@gmail.com> wrote:

Dr Brenda McCaffrey, PhD researcher at Synthesis, is working on a project to develop some dynamical system based tools for modeling human movement.

Can anyone help with the following query:

I’m converging on an approach for Java simulation of Lyapunov exponents based on an amazing set of programs created by a gentleman named Jacob Perkins (https://github.com/thedatachef/sounder/tree/master/udf/src/main/java/sounder/pig/chaos) that he developed using the Sprott algorithm.  This is really deep stuff and uses Hadoop and pig as well as Java.  Do we have resources in these areas?  A couple of hours with an expert would change my life!

Xin Wei

"The weirdest people in the world?" Henrich, Heine, Norenzayan (2010)

A classic paper in BEHAVIORAL AND BRAIN SCIENCES (2010), cautioning any research based on human-subjects studies in design, engineering, and social sciences.  Good for our AME PhD methods course.


The weirdest people in the world? 

Joseph Henrich 
Steven J. Heine
Ara Norenzayan 
Department of Psychology, University of British Columbia, Vancouver

Abstract: 
Behavioral scientists routinely publish broad claims about human psychology and behavior in the world’s top journals based on samples drawn entirely from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies. Researchers – often implicitly – assume that either there is little variation across human populations, or that these “standard subjects” are as representative of the species as any other population. Are these assumptions justified? Here, our review of the comparative database from across the behavioral sciences suggests both that there is substantial variability in experimental results across populations and that WEIRD subjects are particularly unusual compared with the rest of the species – frequent outliers. The domains reviewed include visual perception, fairness, cooperation, spatial reasoning, categorization and inferential induction, moral reasoning, reasoning styles, self-concepts and related motivations, and the heritability of IQ. The findings suggest that members of WEIRD societies, including young children, are among the least representative populations one could find for generalizing about humans. Many of these findings involve domains that are associated with fundamental aspects of psychology, motivation, and behavior – hence, there are no obvious a priori grounds for claiming that a particular behavioral phenomenon is universal based on sampling from a single subpopulation. Overall, these empirical patterns suggests that we need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity. We close by proposing ways to structurally re-organize the behavioral sciences to best tackle these challenges.

plus

Anthony Brandt (Music, Rice), gave a great talk at the Neuroscience of art, innovation and creativity conference dismantling some sweeping claims about universals in musical aesthetic experience and music.

Re: [Synthesis] "The weirdest people in the world?" Henrich, Heine, Norenzayan (2010)

This looks fantastic. Thanks for sharing!

From: <synthesis-research@googlegroups.com> on behalf of "xinwei@mindspring.com" <shaxinwei@gmail.com>
Date: Thursday, August 4, 2016 at 8:58 AM
To: Ron Broglio <rbroglio@earthlink.net>, Grisha Coleman <Grisha.Coleman@asu.edu>, Todd Ingalls <Todd.Ingalls@asu.edu>, Loren Olson <Loren.Olson@asu.edu>, Pavan Turaga <pturaga@asu.edu>, Christian Ziegler <Christian.Ziegler@asu.edu>, Kimberlee Swisher <kimberlee.swisher@gmail.com>, Dehlia Hannah <dhannah1@asu.edu>, Stacey Kuznetsov <kstace@asu.edu>, Christopher Roberts <cmrober2@asu.edu>, Byron Lahey <Byron.Lahey@asu.edu>, Lauren Hayes <laurensarahhayes@gmail.com>, Jessica Rajko <jessica.rajko@asu.edu>, Steven Tepper <Steven.Tepper@asu.edu>, David Tinapple <david.tinapple@asu.edu>, Edward Finn <edfinn@asu.edu>, Garth Paine <Garth.Paine@asu.edu>, Adam Nocek <Adam.Nocek@asu.edu>, Xin Wei Sha <Xinwei.Sha@asu.edu>, Tamara Underiner <Tamara.Underiner@asu.edu>
Cc: Posthaven Post By Email <post@synthesis.posthaven.com>, "synthesis-research@googlegroups.com" <synthesis-research@googlegroups.com>
Subject: [Synthesis] "The weirdest people in the world?" Henrich, Heine, Norenzayan (2010)

A classic paper in BEHAVIORAL AND BRAIN SCIENCES (2010), cautioning any research based on human-subjects studies in design, engineering, and social sciences.  Good for our AME PhD methods course.


The weirdest people in the world? 

Joseph Henrich 
Steven J. Heine
Ara Norenzayan 
Department of Psychology, University of British Columbia, Vancouver

Abstract: 
Behavioral scientists routinely publish broad claims about human psychology and behavior in the world’s top journals based on samples drawn entirely from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies. Researchers – often implicitly – assume that either there is little variation across human populations, or that these “standard subjects” are as representative of the species as any other population. Are these assumptions justified? Here, our review of the comparative database from across the behavioral sciences suggests both that there is substantial variability in experimental results across populations and that WEIRD subjects are particularly unusual compared with the rest of the species – frequent outliers. The domains reviewed include visual perception, fairness, cooperation, spatial reasoning, categorization and inferential induction, moral reasoning, reasoning styles, self-concepts and related motivations, and the heritability of IQ. The findings suggest that members of WEIRD societies, including young children, are among the least representative populations one could find for generalizing about humans. Many of these findings involve domains that are associated with fundamental aspects of psychology, motivation, and behavior – hence, there are no obvious a priori grounds for claiming that a particular behavioral phenomenon is universal based on sampling from a single subpopulation. Overall, these empirical patterns suggests that we need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity. We close by proposing ways to structurally re-organize the behavioral sciences to best tackle these challenges.

plus

Anthony Brandt (Music, Rice), gave a great talk at the Neuroscience of art, innovation and creativity conference dismantling some sweeping claims about universals in musical aesthetic experience and music.

"The weirdest people in the world?" Henrich, Heine, Norenzayan (2010)

A classic paper in BEHAVIORAL AND BRAIN SCIENCES (2010), cautioning any research based on human-subjects studies in design, engineering, and social sciences.  Good for our AME PhD methods course.


The weirdest people in the world? 

Joseph Henrich 
Steven J. Heine
Ara Norenzayan 
Department of Psychology, University of British Columbia, Vancouver

Abstract: 
Behavioral scientists routinely publish broad claims about human psychology and behavior in the world’s top journals based on samples drawn entirely from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies. Researchers – often implicitly – assume that either there is little variation across human populations, or that these “standard subjects” are as representative of the species as any other population. Are these assumptions justified? Here, our review of the comparative database from across the behavioral sciences suggests both that there is substantial variability in experimental results across populations and that WEIRD subjects are particularly unusual compared with the rest of the species – frequent outliers. The domains reviewed include visual perception, fairness, cooperation, spatial reasoning, categorization and inferential induction, moral reasoning, reasoning styles, self-concepts and related motivations, and the heritability of IQ. The findings suggest that members of WEIRD societies, including young children, are among the least representative populations one could find for generalizing about humans. Many of these findings involve domains that are associated with fundamental aspects of psychology, motivation, and behavior – hence, there are no obvious a priori grounds for claiming that a particular behavioral phenomenon is universal based on sampling from a single subpopulation. Overall, these empirical patterns suggests that we need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity. We close by proposing ways to structurally re-organize the behavioral sciences to best tackle these challenges.

plus

Anthony Brandt (Music, Rice), gave a great talk at the Neuroscience of art, innovation and creativity conference dismantling some sweeping claims about universals in musical aesthetic experience and music.

methodology iterate Boulder NCAR study

Hi Brandon, 

As you develop trust thanks to Melissa (and Linda) 

It’d be very very useful to  observe live working sessions — rather than indirect report thru abstract questionnaires.

situations: either of single users working alone , or ensemble.  (In latter case maybe a work session in which you are participant is best)


• live event, write down your impressions of what was going on, afterward
choice: participate in the event as a user, vs as explicit fly on the wall (after you establish trust and permission from others to do this)

• live event with you as participant, with camera + audio recording the entire room to capture all inter-body activity NOT just the screen, head or hands! We will send video snips to  Satinder Gill @ Cambridge for advice o iterating the method.

Iteratively refining the methodology is key to abductive science.


• User views recording with you and does talk aloud: what was she thinking, what was she / peers doing?  
Easiest way to do this is to recurse: play video on Quicktime Player and simply create a second screen grab movie in Quicktime Player 

These are known and effective  techniques in HCI user  experience / design research.  Let's apply them :)


As for trust, I’d suggest that we position your work NOT to replace the "pro's" work with supercomputer simulations, but as entirely different genus with different intent and distinct features:

INTENT:
(1) exploring general features of models
(2) qualitative speculation, backed by quantitative models
(3) more as an instrument for scaffolding ensemble conversation than for intense individual work

FEATURES:

(1) Realtime 
(2) Sketch , minimum power needed to scaffold live scientific creation rather than compete on resolution.
(3) NOT merely educational toy, but as prop for cognitive research in how knowledge is collectively created in improvisatory but highly structured and skilled ways.

Have fun!
Xin Wei

methodology iterate Boulder NCAR study

Hi Brandon, 

As you develop trust thanks to Melissa (and Linda) 

It’d be very very useful to  observe live working sessions — rather than indirect report thru abstract questionnaires.

situations: either of single users working alone , or ensemble.  (In latter case maybe a work session in which you are participant is best)


• live event, write down your impressions of what was going on, afterward
choice: participate in the event as a user, vs as explicit fly on the wall (after you establish trust and permission from others to do this)

• live event with you as participant, with camera + audio recording the entire room to capture all inter-body activity NOT just the screen, head or hands! We will send video snips to  Satinder Gill @ Cambridge for advice o iterating the method.

Iteratively refining the methodology is key to abductive science.


• User views recording with you and does talk aloud: what was she thinking, what was she / peers doing?  
Easiest way to do this is to recurse: play video on Quicktime Player and simply create a second screen grab movie in Quicktime Player 

These are known and effective  techniques in HCI user  experience / design research.  Let's apply them :)


As for trust, I’d suggest that we post your work NOT to replace but entirely different genus un two ways:


(1) Realtime
(2) SKETCH , minimum power needed to scaffold live scientific creation rather than compete on resolution.
(3) NOT merely educational toy, but as prop for cognitive research, meta-science

Have fun!
Xin Wei