Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add various pruner algorithm for different pruning schedule #2853

Open
tu1258 opened this issue Sep 1, 2020 · 3 comments
Open

Add various pruner algorithm for different pruning schedule #2853

tu1258 opened this issue Sep 1, 2020 · 3 comments

Comments

@tu1258
Copy link

@tu1258 tu1258 commented Sep 1, 2020

What would you like to be added: As title

Why is this needed: All pruning schedule except AGPPruner only support level, L1, L2. While there are FPGM, APoZ, MeanActivation and Taylor, it would be much better if we can choose any pruner with any pruning schedule.

Without this feature, how does current nni work:Still good.

Components that may involve changes: I am not so sure tho.

Brief description of your proposal if any: I look into source code of some pruning schedule and it's like

if self.base_algo == 'level':
            schema = CompressorSchema([{
                'sparsity': And(float, lambda n: 0 < n < 1),
                Optional('op_types'): [str],
                Optional('op_names'): [str],
            }], model, _logger)
        elif self.base_algo in ['l1', 'l2']:
            schema = CompressorSchema([{
                'sparsity': And(float, lambda n: 0 < n < 1),
                'op_types': ['Conv2d'],
                Optional('op_names'): [str]
            }], model, _logger) 

Just add more types into it I guess?
Maybe you guys have tried others but met some bug or the loss exploded or anything.

@chicm-ms
Copy link
Contributor

@chicm-ms chicm-ms commented Sep 2, 2020

@tu1258 Thanks for reporting this issue. Do you have some specific requirement about adding which pruner to which scheduler? Some pruners such as Apoz need to collect activation before using, and TaylorFO needs to collect gradients, they may not fit every scheduler easily. We need to handle this case by case if you have specific requirement.

@tu1258
Copy link
Author

@tu1258 tu1258 commented Sep 2, 2020

@chicm-ms Thanks for the reply. Honestly, I don't plan to use specific pruner or pruning schedule, and that's why I report this issue. I want to try different setting and see which is better. I think the more choice the better. This issue is not only for me, I just feel that it would benefit a lot for many users. If anything, I think at least add fpgm which is the most popular?

@chicm-ms
Copy link
Contributor

@chicm-ms chicm-ms commented Sep 4, 2020

@tu1258 FPGM can be added since it does not need to collect activation/gradient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
2.0
Not started
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.