Skip to content

wrong results during training and testing Overlook module #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
HuizaiVictorYao opened this issue Jun 2, 2023 · 5 comments
Open

wrong results during training and testing Overlook module #10

HuizaiVictorYao opened this issue Jun 2, 2023 · 5 comments

Comments

@HuizaiVictorYao
Copy link

Hi, I was trying to reproduce your work on city->foggy setting, after training source model, everything seems to be ok:

INFO:root:AP for person = 0.2404
INFO:root:AP for rider = 0.3151
INFO:root:AP for car = 0.3397
INFO:root:AP for truck = 0.0629
INFO:root:AP for bus = 0.2125
INFO:root:AP for train = 0.0568
INFO:root:AP for motorcycle = 0.1718
INFO:root:AP for bicycle = 0.2643
INFO:root:Mean AP = 0.2079
INFO:root:tested model: source/faster_rcnn_source_1_20_740.pth
INFO:root:test time: 82.1747s

but as I try to train and test the overlooking module, I got very low losses like:

INFO:root:[epoch  1][iter  170/5930] time cost: 2.384635, lr_faster_rcnn: 1.00e-04
INFO:root:                      fastRCNN loss:0.0010 
INFO:root:                      cons_loss: 0.0000,gw_loss:0.0003,gw_global_loss:0.0001,label_loss:0.0003
INFO:root:                      cons_weight: 0.00,gw_ins_weight:0.10,gw_global_weight:0.10,label_weight:1.00
INFO:root:[epoch  1][iter  180/5930] time cost: 2.385559, lr_faster_rcnn: 1.00e-04
INFO:root:                      fastRCNN loss:0.0009 
INFO:root:                      cons_loss: 0.0000,gw_loss:0.0003,gw_global_loss:0.0001,label_loss:0.0003
INFO:root:                      cons_weight: 0.00,gw_ins_weight:0.10,gw_global_weight:0.10,label_weight:1.00
INFO:root:im_detect: 101/500, detect time:0.036s, nms time:0.003s   
INFO:root:[epoch  1][iter  190/5930] time cost: 2.432270, lr_faster_rcnn: 1.00e-04
INFO:root:                      fastRCNN loss:0.0009 
INFO:root:                      cons_loss: 0.0000,gw_loss:0.0003,gw_global_loss:0.0002,label_loss:0.0004
INFO:root:                      cons_weight: 0.00,gw_ins_weight:0.10,gw_global_weight:0.10,label_weight:1.00
INFO:root:[epoch  1][iter  200/5930] time cost: 2.446995, lr_faster_rcnn: 1.00e-04
INFO:root:                      fastRCNN loss:0.0009 
INFO:root:                      cons_loss: 0.0000,gw_loss:0.0004,gw_global_loss:0.0001,label_loss:0.0003
INFO:root:                      cons_weight: 0.00,gw_ins_weight:0.10,gw_global_weight:0.10,label_weight:1.00

after all iters, the results was weird:

INFO:root:VOC07 metric? Yes
INFO:root:AP for person = 0.0004
INFO:root:AP for rider = 0.0000
INFO:root:AP for car = 0.0003
INFO:root:AP for truck = 0.0000
INFO:root:AP for bus = 0.0000
INFO:root:AP for train = 0.0000
INFO:root:AP for motorcycle = 0.0000
INFO:root:AP for bicycle = 0.0001
INFO:root:Mean AP = 0.0001

Do you have similar experience about this or have you got some methods to fix this?

I only have 3090 recourses so I changed pytorch and cuda together with cuda>=11:

>>> torch.__version__
'1.8.0+cu111'

my python version is 3.9.15

and due to the error below:

ERROR:root:Error(s) in loading state_dict for vgg16:
        Missing key(s) in state_dict: "RCNN_base1.0.weight", "RCNN_base1.0.bias", "RCNN_base2.0.weight", "RCNN_base2.0.bias", "RCNN_base2.3.weight", "RCNN_base2.3.bias", "RCNN_base3.0.weight", "RCNN_base3.0.bias", "RCNN_base3.3.weight", "RCNN_base3.3.bias", "RCNN_base4.0.weight", "RCNN_base4.0.bias", "RCNN_base4.2.weight", "RCNN_base4.2.bias", "RCNN_base4.5.weight", "RCNN_base4.5.bias", "RCNN_base4.7.weight", "RCNN_base4.7.bias", "RCNN_base4.9.weight", "RCNN_base4.9.bias", "RCNN_base4.12.weight", "RCNN_base4.12.bias", "RCNN_base4.14.weight", "RCNN_base4.14.bias", "RCNN_base4.16.weight", "RCNN_base4.16.bias". 
        Unexpected key(s) in state_dict: "RCNN_base.0.weight", "RCNN_base.0.bias", "RCNN_base.2.weight", "RCNN_base.2.bias", "RCNN_base.5.weight", "RCNN_base.5.bias", "RCNN_base.7.weight", "RCNN_base.7.bias", "RCNN_base.10.weight", "RCNN_base.10.bias", "RCNN_base.12.weight", "RCNN_base.12.bias", "RCNN_base.14.weight", "RCNN_base.14.bias", "RCNN_base.17.weight", "RCNN_base.17.bias", "RCNN_base.19.weight", "RCNN_base.19.bias", "RCNN_base.21.weight", "RCNN_base.21.bias", "RCNN_base.24.weight", "RCNN_base.24.bias", "RCNN_base.26.weight", "RCNN_base.26.bias", "RCNN_base.28.weight", "RCNN_base.28.bias".

I modified line 20 of Overlook/utils.py like below:

# fasterRCNN.load_state_dict(state_dict)
fasterRCNN.load_state_dict(state_dict,False)

thanks for your reply in advance :)
@HuizaiVictorYao
Copy link
Author

HuizaiVictorYao commented Jun 5, 2023 •
edited
Loading

Whoops! I got it.
While running original traintest_target.py for Python2 in Python3 env, if you have never modified Overlook/utils.py, an error should occur:

ERROR:root:Traceback (most recent call last):
ERROR:root:  File "/home/LODS/Overlook/utils.py", line 20, in load_dict
ERROR:root:fasterRCNN.load_state_dict(state_dict)
ERROR:root:  File "/home/pai/envs/LODS/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1052, in load_state_dict
ERROR:root:self.__class__.__name__, "\n\t".join(error_msgs)))
ERROR:root:RuntimeError
ERROR:root:: 
ERROR:root:Error(s) in loading state_dict for vgg16:
        Missing key(s) in state_dict: "RCNN_base1.0.weight", "RCNN_base1.0.bias", "RCNN_base2.0.weight", "RCNN_base2.0.bias", "RCNN_base2.3.weight", "RCNN_base2.3.bias", "RCNN_base3.0.weight", "RCNN_base3.0.bias", "RCNN_base3.3.weight", "RCNN_base3.3.bias", "RCNN_base4.0.weight", "RCNN_base4.0.bias", "RCNN_base4.2.weight", "RCNN_base4.2.bias", "RCNN_base4.5.weight", "RCNN_base4.5.bias", "RCNN_base4.7.weight", "RCNN_base4.7.bias", "RCNN_base4.9.weight", "RCNN_base4.9.bias", "RCNN_base4.12.weight", "RCNN_base4.12.bias", "RCNN_base4.14.weight", "RCNN_base4.14.bias", "RCNN_base4.16.weight", "RCNN_base4.16.bias". 
        Unexpected key(s) in state_dict: "RCNN_base.0.weight", "RCNN_base.0.bias", "RCNN_base.2.weight", "RCNN_base.2.bias", "RCNN_base.5.weight", "RCNN_base.5.bias", "RCNN_base.7.weight", "RCNN_base.7.bias", "RCNN_base.10.weight", "RCNN_base.10.bias", "RCNN_base.12.weight", "RCNN_base.12.bias", "RCNN_base.14.weight", "RCNN_base.14.bias", "RCNN_base.17.weight", "RCNN_base.17.bias", "RCNN_base.19.weight", "RCNN_base.19.bias", "RCNN_base.21.weight", "RCNN_base.21.bias", "RCNN_base.24.weight", "RCNN_base.24.bias", "RCNN_base.26.weight", "RCNN_base.26.bias", "RCNN_base.28.weight", "RCNN_base.28.bias". 
        
ERROR:root:
During handling of the above exception, another exception occurred:
ERROR:root:Traceback (most recent call last):
ERROR:root:  File "traintest_target.py", line 1179, in <module>
ERROR:root:train(args,models,args.gpu_default_id)
ERROR:root:  File "traintest_target.py", line 935, in train
ERROR:root:load_dict(fasterRCNN, checkpoint['model'])
ERROR:root:  File "/home/LODS/Overlook/utils.py", line 27, in load_dict
ERROR:root:state_dict[det[i]] = state_dict.pop(ori[i])
ERROR:root:TypeError
ERROR:root:: 
ERROR:root:'odict_keys' object does not support indexing

I guess that's some kind of deprecated method. So I look into the first error:Missing key(s) in state_dict:... Unexpected key(s) in state_dict... and modified load_state_dict as follow:

# fasterRCNN.load_state_dict(state_dict)
fasterRCNN.load_state_dict(state_dict,False)

However, due to the huge difference between python2 and python3, the difference between two vgg16 dicts is not only the difference between layer names but also the data structure of ordered directory, so the modification above actually didn't load pre-trained models to our models correctly.

As I refer to utils.py, I noticed a try-except in load_dict:

def load_dict(fasterRCNN, state_dict):
    try:
        fasterRCNN.load_state_dict(state_dict)
    except:
        ori = state_dict.keys()
        det = fasterRCNN.state_dict().keys()
        j=0
        for i in range(len(ori)):
            j+=1
            state_dict[det[i]] = state_dict.pop(ori[i])
        fasterRCNN.load_state_dict(state_dict)
        del ori
        del det

I finally realized that if load_state_dict failed without "Strict=False", there is also an except, and the codes in except also failed because the second part error 'odict_keys' object does not support indexing. I looked into this error and modified following the method of [THIS LINK](https://stackoverflow.com/questions/39284842/order-dictionary-index-in-python) as below:

def load_dict(fasterRCNN, state_dict):
    try:
        fasterRCNN.load_state_dict(state_dict)
    except:
    	# ori = state_dict.keys()
        ori = list(state_dict.keys())
        # det = fasterRCNN.state_dict().keys()
        det = list(fasterRCNN.state_dict().keys())
        j=0
        for i in range(len(ori)):
            j+=1
            state_dict[det[i]] = state_dict.pop(ori[i])
        fasterRCNN.load_state_dict(state_dict)
        del ori
        del det

It finally worked:

INFO:root:VOC07 metric? Yes
INFO:root:AP for person = 0.2505
INFO:root:AP for rider = 0.3318
INFO:root:AP for car = 0.3433
INFO:root:[epoch  1][iter  400/5930] time cost: 3.399532, lr_faster_rcnn: 1.00e-09
INFO:root:                      fastRCNN loss:0.0605 
INFO:root:                      cons_loss: 0.00000000,gw_loss:0.00519128,gw_global_loss:0.00055907,label_loss:0.12242991
INFO:root:                      cons_weight: 0.00,gw_ins_weight:0.10,gw_global_weight:0.10,label_weight:1.00
INFO:root:AP for truck = 0.0671
INFO:root:AP for bus = 0.2095
INFO:root:AP for train = 0.0968
INFO:root:AP for motorcycle = 0.1642
INFO:root:AP for bicycle = 0.2688
INFO:root:Mean AP = 0.2165

Thanks for your excellent work and tell me to close this issue if you want to!

@HuizaiVictorYao
Copy link
Author

HuizaiVictorYao commented Jun 5, 2023 •
edited
Loading

However the result is still different from the paper as #7 pointed out since I used python3.
I do only have 3090 resources. It seems that 3090 doesn't match the version of python2.7 and python2.7's highest supported version of cuda and pytorch.

@simvarail
Copy link

However the result is still difficult from the paper as #7 pointed out since I used python3. I do only have 3090 resources. It seems that 3090 doesn't match the version of python2.7 and python2.7's highest supported version of cuda and pytorch.

I'm only achieving a 24.24 mAP, which falls far short of the 35.8 mAP advertised in the paper. It's strange that even though I'm running Python 3 instead of 2, the difference is so noticeable. This seems to be more pronounced in the Cityscape dataset compared to others, but I'm unsure how to explain or correct this discrepancy. When evaluating the Pascal to Clipart dataset, the results are relatively close to the advertised mAP.

@HuizaiVictorYao
Copy link
Author

However the result is still difficult from the paper as #7 pointed out since I used python3. I do only have 3090 resources. It seems that 3090 doesn't match the version of python2.7 and python2.7's highest supported version of cuda and pytorch.

I'm only achieving a 24.24 mAP, which falls far short of the 35.8 mAP advertised in the paper. It's strange that even though I'm running Python 3 instead of 2, the difference is so noticeable. This seems to be more pronounced in the Cityscape dataset compared to others, but I'm unsure how to explain or correct this discrepancy. When evaluating the Pascal to Clipart dataset, the results are relatively close to the advertised mAP.

I've given up working on this repo now. I'm considering renting GPUs to build the same environment as the author claimed if I find it really necessary for my future work. By the way, I can reproduce normal mAPs in my environment using this repo, on which this LODS repo is based.

@simvarail
Copy link

However the result is still difficult from the paper as #7 pointed out since I used python3. I do only have 3090 resources. It seems that 3090 doesn't match the version of python2.7 and python2.7's highest supported version of cuda and pytorch.

I'm only achieving a 24.24 mAP, which falls far short of the 35.8 mAP advertised in the paper. It's strange that even though I'm running Python 3 instead of 2, the difference is so noticeable. This seems to be more pronounced in the Cityscape dataset compared to others, but I'm unsure how to explain or correct this discrepancy. When evaluating the Pascal to Clipart dataset, the results are relatively close to the advertised mAP.

I've given up working on this repo now. I'm considering renting GPUs to build the same environment as the author claimed if I find it really necessary for my future work. By the way, I can reproduce normal mAPs in my environment using this repo, on which this LODS repo is based.

Thank you for your reply. I appreciate it. I'll definitely give it a try, although I must admit that my primary interest lies in source-free domain adaptation for object detection. By the way, if you happen to know of any other methods that have achieved promising results and come with working code, I would greatly appreciate it if you could share them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@simvarail @HuizaiVictorYao