If the hypothesis is that an AGI-level AI can accomplish a meaningful part of economically valuable work on its own without needing human expertise, it seems extremely unlikely that any such system would be released open weights. The benefits of controlling use would be too high. - ThreadSky

emollick.bsky.social • 19 days ago

If the hypothesis is that an AGI-level AI can accomplish a meaningful part of economically valuable work on its own without needing human expertise, it seems extremely unlikely that any such system would be released open weights.

The benefits of controlling use would be too high.

Comments

benmcmahan.bsky.social•19 days ago

My hope is this relies on the assumption that money is the only motivator in the world and that that isn't actually true. Someone will release open weights for some philosophical belief.

asmeurer.com•19 days ago

Instead of trying armchair reason about this from first principles, look at actual open source software, which doesn't match your naive intuition.

nsaphra.bsky.social•19 days ago

Why wouldn't they? There really is no moat, so may as well be the one who released it rather than another obscure player in the market.

togelius.bsky.social•19 days ago

This assumes that there is a sharp boundary between "AGI-level" and "not AGI-level". So far, AI progress has been gradual, so this seems unlikely. Also, there will still be competition among multiple actors. So it would still make sense to open-source.

tedunderwood.me•19 days ago

Yes. Everything we've seen over the last two years suggests that "general intelligence" is a fuzzy problem that will get solved piecemeal, not a crisp boundary.

Models could explain quantum mechanics eloquently way before they could count the number of r's in strawberry.

davidmanheim.alter.org.il•19 days ago

That works as a strong reason for the very first such system not to be open-sourced, but the logic seems to fail almost immediately afterwards.

moultano.bsky.social•19 days ago

Unless they don't know it until after the release. Sampling, scaffolding, or fine tuning can upgrade a model afterwards.

emollick.bsky.social•19 days ago

The argument for releasing open weights models has always been a bit tenuous compared to the equivalent argument for open source software (where value capture for ancillary services is more clear), but it would break down completely if the economic value of owning a model goes up

watsonix.bsky.social•19 days ago

I believe this would have to assume that such a product would be perfect and that workers that use it would be perfect. Otherwise, companies want support, customization, additions etc. The same reasons that complex open source software can still make money.

dtjw.bsky.social•19 days ago

Our current power seeking tendencies makes it unlikely that we would intentionally release AGI-level AI.

There is a non-zero chance that AGI-level AI may be created unintentionally by automated AI systems by simply combining various research paper found online.

trthcykr.bsky.social•19 days ago

There are many idealists in this field. So there is a serious chance someone might do it.

watsonix.bsky.social•19 days ago

The immediately adjacent example of open source software would seem to call this statement into question. Where does the "extremely unlikely" part come from with that taken into account?

Comments

Posting Rules

Reply