Adding DS Feature API in accelerator

[duli2012]

This PR is a prototype of adding API for capabilities in accelerators including:

1. define capabilities in abstract_accelerator
2. set capabilities in cuda_accelerator

Welcome hardware vendors to define capabilities for their own hardware.

[tjruwase]

These hardcoded constants, such as "zero1", should be defined as symbolic constants in a global location.

[duli2012]

@tjruwase As we discussed, I collected the names in *OpBuilder and put it in constants.py. Do we need add other features like optimizers as ds features too?

[delock]

Hi @duli2012 thanks for adding this interface. I have always been worring accelerator interface# may grow too big when we propose more and more capabilities into it, this interface is a good way to put all capabilities into one interface. My comments below:

1. For zero1/2/3, it sounds like accelerator agnostic features. Basically it should be supported if accelerator runtime and communication collectives is well implemented.
2. For sparse_attn, it is in the category of whether certain OpBuilder is implemented. Previous way of telling whether ops is implemented is through compatible ops as the following link. I think its better to move this capability group into capability interface since the new interfacve look more intuitive to use. Is it possible to make it auto reflect accelerator OpBuilder implementation so there will be less maintentance work? A minor suggestion is name capability in this group with same prefix i.e. op.sparse_attn
   https://github.com/microsoft/DeepSpeed/blob/b22706a7211366abf2df98a0d118ea1d3a837e21/tests/unit/inference/test_inference.py#L336
3. For 1-bit adam, I agree this is a case that could be covered by this interface.

What should we do with already existing accelerator interface that falls into category of capabilities? Should they be added to the new inferface or just keep them that way? I think that's an open to discuss.

[duli2012]

Hi @duli2012 thanks for adding this interface. I have always been worring accelerator interface# may grow too big when we propose more and more capabilities into it, this interface is a good way to put all capabilities into one interface. My comments below:

1. For zero1/2/3, it sounds like accelerator agnostic features. Basically it should be supported if accelerator runtime and communication collectives is well implemented.
2. For sparse_attn, it is in the category of whether certain OpBuilder is implemented. Previous way of telling whether ops is implemented is through compatible ops as the following link. I think its better to move this capability group into capability interface since the new interfacve look more intuitive to use. Is it possible to make it auto reflect accelerator OpBuilder implementation so there will be less maintentance work? A minor suggestion is name capability in this group with same prefix i.e. op.sparse_attn
   https://github.com/microsoft/DeepSpeed/blob/b22706a7211366abf2df98a0d118ea1d3a837e21/tests/unit/inference/test_inference.py#L336
3. For 1-bit adam, I agree this is a case that could be covered by this interface.

What should we do with already existing accelerator interface that falls into category of capabilities? Should they be added to the new inferface or just keep them that way? I think that's an open to discuss.

Thanks @delock for your comments. I integrated most of them. please take a look.

[rogerxfeng8]

A general name is needed to cover the non-cuda devices?

[loadams]

Hi @rogerxfeng8 - is there a specific one you're referencing? I believe we call all devices (cuda and non-cuda) accelerators.

[delock]

Thanks @duli2012 , my intuition is zero 1/2/3 should not among accelerator feature list. Zero stage code is shared between different accelerators and there is no interface specific to zero stage, so I wonder what makes an accelerator not support zero features?

For OP related features i.e. OP_ASYNC_IO, etc. I think there needs to be an mechanism sync them with opbuilder implementation state automatically, otherwise there will be manual maintenance cost each time a new op is introduced.

[delock]

This reflection mechanism better be lazy initialized. Otherwise there might be circular dependence because this init function be called before __compatible_ops__ being initialized.