On the one hand all the hype people are right for robotics (ml and ai are the future, old model based methods are largely going away) but I'm still optimistic about *modular* architectures and 3d priors over some of the pure end to end stuff, mostly because I think it helps generalization - ThreadSky

cpaxton.bsky.social • 5 days ago

On the one hand all the hype people are right for robotics (ml and ai are the future, old model based methods are largely going away) but I'm still optimistic about *modular* architectures and 3d priors over some of the pure end to end stuff, mostly because I think it helps generalization

Comments

basavasagar18.bsky.social•4 days ago

I think pure end-to-end learning will be a deployment scenario. Most of the model based planning methods and priors will be used for data generation in simulation for efficient exploration in data collection. But it is hard to win over pure end-to-end learning system if you have the data once.

cpaxton.bsky.social•3 days ago

Modularity and end to end learning aren't opposites. I could have a system composed of pre-trained, specialized components which I then optimize/fine-tune end to end. I'm against undifferentiated blobs of neural net here, massive transformer stacks dont feel appropriate for robotics

cpaxton.bsky.social•3 days ago

I agree though about learning vs model based

basavasagar18.bsky.social•3 days ago

Well, is it efficient, yes! But is it the optimal, probably no. The problem with pre-trained modules is that their representations are biased by their objective to fit a benchmark. And this objective encapsulates years of community based benchmark overfitting that we don't even think of biases ..

cpaxton.bsky.social•3 days ago

I think this is true but it's still easier to do it separately

basavasagar18.bsky.social•3 days ago

Yes, I agree. I am thinking of a distinct future when we have all the resources in our solar system. Lol, too hypothetical!

basavasagar18.bsky.social•3 days ago

Example the bias of CNNs to exploit high-frequency components in image might be good for CV benchmark performance, if you feed the same representations to a robotic policy then it is clearly sub-optimal, shape is more important here ....

basavasagar18.bsky.social•3 days ago

Fine-tuning is a fine game when you know the test-distribution and I believe that active fine-tuning at test time will be on the cards. But none can truly generalize out-of-distribution. You either collect a hell lot of data or stumble upon an architecture that truly generalizes of OOD for your task

thermans.bsky.social•5 days ago

Planning generalizes probably. Yes you need models, it can be expensive, and reactivity can be hard, but we should use it where we can. It’s always funny to me when folks argue for learning over planning for generalization. Yes we need to learn, but planning isn’t whats stopping generalization.

cpaxton.bsky.social•5 days ago

Yeah definitely. Lots of things it doesn't make any real sense to learn

thermans.bsky.social•5 days ago

This was supposed to say “provably” not probably. Baby slapping the screen while typing is the likely culprit here.

isaiahbishop.bsky.social•5 days ago

I think this is the waymo method right? One of the reasons they can do things safely

isaiahbishop.bsky.social•5 days ago

there was a good debrief from the failed truck startup otto with the same perspective. If you want safety and control cant really do that kind of end-to-end magic

cpaxton.bsky.social•5 days ago

I think these days waymo does more and more e2e stuff. Which imo is good; a good modular stack can be trained e2e too.

Also companies like otto failed because they went way too far in the other direction. Classic robotics clearly does not scale

isaiahbishop.bsky.social•5 days ago

Yeah i think theres going to be a good balance, and you can probably just pull a good modular system out of an end to end model tbh

somerobot0.bsky.social•5 days ago

These seem like good methods for bootstrapping, why not tell the ml system things you can be sure about

cpaxton.bsky.social•5 days ago

How much data do you need to learn an A* planner? The answer generally is huge

somerobot0.bsky.social•5 days ago

Right! one answer to that now is to have a big foundation model you’ve trained with accumulating data and then fine tune in your deployment/domain/task.

You were saying you don’t think the scaling law is there for the VLA approaches though, right?

cpaxton.bsky.social•5 days ago

I've actually written a bunch on substack about scaling laws

cpaxton.bsky.social•5 days ago

I dont think we have the recipe cracked. Scaling laws are... not real laws

somerobot0.bsky.social•5 days ago

It has been very exciting to see the recipe get closer in the last few years. One way is to go sim2real + demonstrations to get there, but are you talking more about neuro symbolic or another way to incorporate priors?

Comments

Posting Rules

Reply