EXPERIMENTATION LOG — Motion tracking, hand objects

Aim

Experiment with poseNet, how I might map it to the ortho() perspective, how I could bring in objects and play with the input of a body. 

Bring in visual references — acupuncture charts, fingernail guards. 

Precedents / context

Feedback from Andrew to experiment with poseNet, body/gesture. 

Process / methods

• Imported the ml5 library in and poseNet. 
• Got poseNet code off an example online, by Kyle McDonald: https://editor.p5js.org/kylemcdonald/sketches/H1OoUd9h7
  
• First task was to get it working in the 3D, ortho() perspective because that's the space of the theatre. At first I was using quite a bit of rotate which i think made things unnecessarily complicated. 
  
• Loaded a hand model with the objective to track it to your wrist points. 

• Got the keypoints and skeleton situated in 3D and ortho()
• Imported in the scan of the acupunture chart as a possible image element/texture for the object. 

• Maybe I could do visual experimentation around these visuals — illustration, collage, photoshop
• Just tried using it as a texture for the vases — which looks kind of cool but does not show/allude to the content in any way. 
  
• Not sure if I like the vases? They're a nice framing device, and an interesting alternative to a very literal stage setup like I have already but it doesn't feel intentional enough. 

• Still had keypoints/skeleton up but realised I couldn't attach objects to keypoints (or can I???? Try making the translate values keypoint varia) so tried mapping the x and y coordinates of the wrists to the colour/light values. 

• Feel like I can do a lot more experimentation with type, had a quick go at making those paper squares that say 'Fú' 福, or good fortune. 

Reflection on action

• I think the scale of the hands is effective visually, you almost feel as though you're being hugged or embraced by them. The colour mapping is cool. I think it looks good but is perhaps not that interesting? It needs to tie in to the concept better. 
• 
  
• Am really pleased that I got the motion tracking to work in 3D and ortho(), although I suspect I was making it much harder for myself at first with the rotate? 
  
  However I don't feel like I'm making the most of the motion interaction. The imagined plan was to track the different points of the hand, so it could be a lot more gestural, but poseNet has only been trained on key joints, so hand/arm-wise I'd have the wrists, elbows and shoulders to work with. 
  
  I know it does the whole body, and from my testing it seems to work really well, but as a website you access through a computer device (laptops and desktops), limiting the scope to the chest up and arms seems like the best choice.  

• The Vase and type/good fortune character don't feel intentional and authentic enough. It feels like a surface-level, rudimentary attempt to make something Chinese. Same with the gradient/light changing material of the hands. It feels too obviously queer, without getting to the heart of the experience of queerness. Like, oh, of course, rainbow, gradients. That's almost too easy. 
  
  I think what worked so well with the poetry translations is that they feel really personal and thought through, and they were specific. As a cultural "object", they were structurally and tonally different from traditional western poetry. It just has more depth, so that the representation feels more earned and materialised in a way that it not always is. 
  
  I just feel like I need to go deeper into the experience of this intersection — what it feels and means to live it and be it. How it actually affords me possibilities and enriches my perspective/experience of the world? How that might have come from a place of shame/pain but be reframed and reclaimed. I need to earn reclaiming. It comes with a process and reflection/iteration. 

• The different objects and looser layout/setup of this stage makes me think of flashgame environments? Particularly with the ortho() perspective. It could be really fun to play with this idea and the playfulness of a theatre environment with props (other google poly models that I've seen around — teapots, bamboo stalks, type/characters, costumes? Pick things up, interact with them, move through the stage... 

Reflection for action

• I think I absolutely need to sketch and work through different possibilities. 
• Put it to the slack group again — more gestural movements. 
• Still feel like there's a lot more to be explored with typography and image making. I think that's a good thing to get stuck into during stuvac / the next two weeks. A different kind of making to all the coding. 

References

https://editor.p5js.org/kylemcdonald/sketches/H1OoUd9h7

https://poly.google.com/view/fLzSqhcYmKg