I spent quite a lot of time at Dynamicland Oakland. It's great fun and the people are just wonderful. I think of Realtalk as adding another layer to programs, the Physical Layer. Program behavior can be determined and controlled by physical layout as well as code. It's not completely unlike a frontend/backend distinction, and in fact you could make a Realtalk simulator using a javascript canvas much more easily than building the camera/projector setup.
Just like building full stack software, there's a large amount of nonobvious skill in dictating your separations of concerns between the physical layer and software layer. Good programs are flexible, remixable, modular, intuitive, and let non-programmers make nontrivial interactions and enhancements via the physical layer. Bad programs require you to have the physical objects in a particular configuration, or break completely if one piece of paper is lost. I found these programming design questions a really interesting part of playing at Dynamicland.
A solid limitation of the system is that the pieces of programs aren't actually modular. You can't take a Cat from one program and a Dog from another program and have the dog interact with the cat. This is obvious in software - that's why we design APIs - but it's frustrating when all your programs exist in the same space (that's the whole selling point) and when bringing part of Program A into Program B is so intuitive and, when you have dozens of these programs lying around the room, inevitable.
I'd love to see them explore (wait for it...) using AI. Incorporating object recognition could remove the need for pasting dots onto every object by defining rules like "when you see a car, color it Red". It could allow for inter-program interoperability via the shared language of object recognition. And it could even determining logical interactions in a fun and surprising way: what _should_ happen when I take the cactus from this program and put it on top of the balloon from that program?
> You can't take a Cat from one program and a Dog from another program and have the dog interact with the cat. This is obvious in software - that's why we design APIs - but it's frustrating when all your programs exist in the same space
I also thought this, but learned that what I needed was an Alan Kay style "change of perspective".
The key for me was understanding the "wish" and "claim" concepts in Realtalk. In your example above, you would need to separate your "Dog" program into two programs: One that was simply a program claiming "I am a dog" and another that would encode dog behavior, something like "I wish that a bark sound would play when a dog is near a house" - then you'd leave it up to Realtalk to make that happen. Adding "Cat" behavior would mean adding two new programs: one to claim "I am a cat" and another "I wish that a meow sound would play when a cat is near a house". To make the "Dog" and "Cat" interact, you'd add a program that said "I wish that a growl sound would play when a dog is near a cat" and so on.
Another example that might help is how I learned this myself: I made a set of playing cards. What I ended up with was 52 pieces of card stock, each one had a program that was simply "Claim that I am card X" - and then I made separate programs the give those cards meaning. For example a program to give the cards a style would be something like "I wish that card 1 will have the Ace of Spaces printed on it, etc" and another program would be something like "I wish that the sum of all face values of the cards on this line is printed next to the line"
It took me several days to internalize this, but once I did things started getting fun pretty quickly. For example, I was able to make a program to "clone" a real world image onto a playing card by having Realtalk take a picture of a rectangle and then always project that image onto a particular card when it was face up. Because the cards just made claims about their identities, this let me separate the designs on the cards from the rules as well as add "training mode" programs to help teach basics. And they were all decoupled!
it should be possible to make objects that are general-purpose, you end up with a bunch of low-code reusable things like "dog" or "car", and they have stable representations than can be used as data in multiple other programs
> I'd love to see them explore (wait for it...) using AI.
Well, you know what, whenever someone invents such a thing, it would be great. But it doesn't exist yet and there's good reason to think it never will.
What is 100% certain is that LLMs are not intelligent and never will be. They are just very good at producing totally stupid, mindless output that convinces some people it's real and was produced by humans.
Just like building full stack software, there's a large amount of nonobvious skill in dictating your separations of concerns between the physical layer and software layer. Good programs are flexible, remixable, modular, intuitive, and let non-programmers make nontrivial interactions and enhancements via the physical layer. Bad programs require you to have the physical objects in a particular configuration, or break completely if one piece of paper is lost. I found these programming design questions a really interesting part of playing at Dynamicland.
A solid limitation of the system is that the pieces of programs aren't actually modular. You can't take a Cat from one program and a Dog from another program and have the dog interact with the cat. This is obvious in software - that's why we design APIs - but it's frustrating when all your programs exist in the same space (that's the whole selling point) and when bringing part of Program A into Program B is so intuitive and, when you have dozens of these programs lying around the room, inevitable.
I'd love to see them explore (wait for it...) using AI. Incorporating object recognition could remove the need for pasting dots onto every object by defining rules like "when you see a car, color it Red". It could allow for inter-program interoperability via the shared language of object recognition. And it could even determining logical interactions in a fun and surprising way: what _should_ happen when I take the cactus from this program and put it on top of the balloon from that program?