088: GenAI gets physical

If it's a pattern, a machine can recognize and replicate it

First, I’m not attending the 57th annual Consumer Electronics Show (CES) but my friend Greg is, and he covers these kinds of endeavors with glee and useful insight. Look for updates starting this evening.

Second, Dan Hon has written very thoughtfully about the OpenAI v The New York Times lawsuit. (Scroll down to “Impossible AI.”) Gary Marcus has also been researching and writing about AI and copyright.

---

The shrimp flipping pattern

On a broad level, so much of human living is pattern after pattern after pattern after pattern, e.g. putting dishes into the dishwasher, or me and that box of Costco chocolate chip cookies—just a pattern! One of the outcomes of GenAI this past year has been the realization much of writing, designing and art-making is rooted in well worn patterns, which an LLM can learn from and then re-create (really quickly). I’m not talking about taste, or quality here—or even “creativity,” just mechanics.

99% of what I’ve been writing about GenAI the past year has been focused on words and images, and mostly on screens, now what if we take the pattern-reading and replication of GenAI and apply it to physical tasks?

My six year old son has a condition called Arthrogryposis which affects joints in a body, and shows up (or “presents”) in a wide variety of outcomes. In his case, Felix’s elbow joints can’t bend. His jaw is also constrained somewhat by the condition. Long story short: Felix can’t feed himself; he is tube fed.

But couldn’t a robot learn to repeat the patterns of his arm movements to facilitate mouth feeding?

(The answer is “yes.”)

The technology in this video is called Obi. It’s in a long line of robots learning to mimic humans and animals (hello, Boston Scientific dog gymnast robot!). Felix got to demo Obi at his physical therapy last week. Pretty cool! Then on Monday I read this paper about Mobile ALOHA, an LLM-infused research project at Stanford, which expands on experiences like Felix’s.

Solid recaps on Mobile ALOHA via The Neuron and especially Jack Clark from Import AI. As Clark writes, “Robots may be nearing their ‘imagenet moment’ when both the cost of learning robot behaviors falls, as does the data for learning their behaviors. Mobile ALOHA makes it way cheaper to collect data for robot behaviors and also to learn on real world platforms.”

I think most of us expected the first big wave of “robots replacing human tasks” to be like the Obi and Mobile ALOHA examples, and not ChatGPT writing poems. But maybe replicating word patterns was just technically easier than replicating shrimp flipping. They’re all just patterns!

---

AI+Creativity Update

🤓 File under: “The next Newton or iPod?” The Apple Vision Pro supply chain is cranking into production. Come February, we’ll begin to see if, how and why augmented reality is ready for prime time on our faces. I’m especially curious to see GenAI applications leveraging spatial computing.

👚 File under: “Well, of course.” The motivation—24% of clothes ordered online are returned—is significant, notes TechCrunch. And what are clothes but patterns? Good for Amazon to bake various incarnations of GenAI into online clothing shopping if for no other reason than to reduce returns.

🎨 I can’t get enough of AI artist Kelly Boesch. Discovered her via TikTok. She’s also on Instagram. Stunning!