RuntimeError: Error(s) in loading state_dict for TokenConditionedTransformer

[trinhtuanvubk]

I just run the infer file and got this eror:

Traceback (most recent call last):
  File "/workspace/OPEN-MUSICLM/scripts/infer.py", line 66, in <module>
    musiclm = create_musiclm_from_config(
  File "/workspace/OPEN-MUSICLM/scripts/../open_musiclm/config.py", line 442, in create_musiclm_from_config
    semantic_transformer = create_semantic_transformer_from_config(model_config, semantic_path, device)
  File "/workspace/OPEN-MUSICLM/scripts/../open_musiclm/config.py", line 258, in create_semantic_transformer_from_config
    load_model(transformer, checkpoint_path)
  File "/workspace/OPEN-MUSICLM/scripts/../open_musiclm/config.py", line 204, in load_model
    model.load_state_dict(pkg)
  File "/home/user/miniconda/envs/open-musiclm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for TokenConditionedTransformer:
	Unexpected key(s) in state_dict: "transformer.layers.6.0.q_scale", "transformer.layers.6.0.k_scale", "transformer.layers.6.0.norm.gamma", "transformer.layers.6.0.norm.beta", "transformer.layers.6.0.to_q.weight", "transformer.layers.6.0.to_kv.weight", "transformer.layers.6.0.to_out.0.weight", "transformer.layers.6.2.0.gamma", "transformer.layers.6.2.0.beta", "transformer.layers.6.2.1.weight", "transformer.layers.6.2.2.ds_conv.weight", "transformer.layers.6.2.4.gamma", "transformer.layers.6.2.4.beta", "transformer.layers.6.2.6.weight", "transformer.layers.7.0.q_scale", "transformer.layers.7.0.k_scale", "transformer.layers.7.0.norm.gamma", "transformer.layers.7.0.norm.beta", "transformer.layers.7.0.to_q.weight", "transformer.layers.7.0.to_kv.weight", "transformer.layers.7.0.to_out.0.weight", "transformer.layers.7.2.0.gamma", "transformer.layers.7.2.0.beta", "transformer.layers.7.2.1.weight", "transformer.layers.7.2.2.ds_conv.weight", "transformer.layers.7.2.4.gamma", "transformer.layers.7.2.4.beta", "transformer.layers.7.2.6.weight", "transformer.layers.8.0.q_scale", "transformer.layers.8.0.k_scale", "transformer.layers.8.0.norm.gamma", "transformer.layers.8.0.norm.beta", "transformer.layers.8.0.to_q.weight", "transformer.layers.8.0.to_kv.weight", "transformer.layers.8.0.to_out.0.weight", "transformer.layers.8.2.0.gamma", "transformer.layers.8.2.0.beta", "transformer.layers.8.2.1.weight", "transformer.layers.8.2.2.ds_conv.weight", "transformer.layers.8.2.4.gamma", "transformer.layers.8.2.4.beta", "transformer.layers.8.2.6.weight", "transformer.layers.9.0.q_scale", "transformer.layers.9.0.k_scale", "transformer.layers.9.0.norm.gamma", "transformer.layers.9.0.norm.beta", "transformer.layers.9.0.to_q.weight", "transformer.layers.9.0.to_kv.weight", "transformer.layers.9.0.to_out.0.weight", "transformer.layers.9.2.0.gamma", "transformer.layers.9.2.0.beta", "transformer.layers.9.2.1.weight", "transformer.layers.9.2.2.ds_conv.weight", "transformer.layers.9.2.4.gamma", "transformer.layers.9.2.4.beta", "transformer.layers.9.2.6.weight", "transformer.layers.10.0.q_scale", "transformer.layers.10.0.k_scale", "transformer.layers.10.0.norm.gamma", "transformer.layers.10.0.norm.beta", "transformer.layers.10.0.to_q.weight", "transformer.layers.10.0.to_kv.weight", "transformer.layers.10.0.to_out.0.weight", "transformer.layers.10.2.0.gamma", "transformer.layers.10.2.0.beta", "transformer.layers.10.2.1.weight", "transformer.layers.10.2.2.ds_conv.weight", "transformer.layers.10.2.4.gamma", "transformer.layers.10.2.4.beta", "transformer.layers.10.2.6.weight", "transformer.layers.11.0.q_scale", "transformer.layers.11.0.k_scale", "transformer.layers.11.0.norm.gamma", "transformer.layers.11.0.norm.beta", "transformer.layers.11.0.to_q.weight", "transformer.layers.11.0.to_kv.weight", "transformer.layers.11.0.to_out.0.weight", "transformer.layers.11.2.0.gamma", "transformer.layers.11.2.0.beta", "transformer.layers.11.2.1.weight", "transformer.layers.11.2.2.ds_conv.weight", "transformer.layers.11.2.4.gamma", "transformer.layers.11.2.4.beta", "transformer.layers.11.2.6.weight", "transformer.layers.12.0.q_scale", "transformer.layers.12.0.k_scale", "transformer.layers.12.0.norm.gamma", "transformer.layers.12.0.norm.beta", "transformer.layers.12.0.to_q.weight", "transformer.layers.12.0.to_kv.weight", "transformer.layers.12.0.to_out.0.weight", "transformer.layers.12.2.0.gamma", "transformer.layers.12.2.0.beta", "transformer.layers.12.2.1.weight", "transformer.layers.12.2.2.ds_conv.weight", "transformer.layers.12.2.4.gamma", "transformer.layers.12.2.4.beta", "transformer.layers.12.2.6.weight", "transformer.layers.13.0.q_scale", "transformer.layers.13.0.k_scale", "transformer.layers.13.0.norm.gamma", "transformer.layers.13.0.norm.beta", "transformer.layers.13.0.to_q.weight", "transformer.layers.13.0.to_kv.weight", "transformer.layers.13.0.to_out.0.weight", "transformer.layers.13.2.0.gamma", "transformer.layers.13.2.0.beta", "transformer.layers.13.2.1.weight", "transformer.layers.13.2.2.ds_conv.weight", "transformer.layers.13.2.4.gamma", "transformer.layers.13.2.4.beta", "transformer.layers.13.2.6.weight", "transformer.layers.14.0.q_scale", "transformer.layers.14.0.k_scale", "transformer.layers.14.0.norm.gamma", "transformer.layers.14.0.norm.beta", "transformer.layers.14.0.to_q.weight", "transformer.layers.14.0.to_kv.weight", "transformer.layers.14.0.to_out.0.weight", "transformer.layers.14.2.0.gamma", "transformer.layers.14.2.0.beta", "transformer.layers.14.2.1.weight", "transformer.layers.14.2.2.ds_conv.weight", "transformer.layers.14.2.4.gamma", "transformer.layers.14.2.4.beta", "transformer.layers.14.2.6.weight", "transformer.layers.15.0.q_scale", "transformer.layers.15.0.k_scale", "transformer.layers.15.0.norm.gamma", "transformer.layers.15.0.norm.beta", "transformer.layers.15.0.to_q.weight", "transformer.layers.15.0.to_kv.weight", "transformer.layers.15.0.to_out.0.weight", "transformer.layers.15.2.0.gamma", "transformer.layers.15.2.0.beta", "transformer.layers.15.2.1.weight", "transformer.layers.15.2.2.ds_conv.weight", "transformer.layers.15.2.4.gamma", "transformer.layers.15.2.4.beta", "transformer.layers.15.2.6.weight", "transformer.layers.16.0.q_scale", "transformer.layers.16.0.k_scale", "transformer.layers.16.0.norm.gamma", "transformer.layers.16.0.norm.beta", "transformer.layers.16.0.to_q.weight", "transformer.layers.16.0.to_kv.weight", "transformer.layers.16.0.to_out.0.weight", "transformer.layers.16.2.0.gamma", "transformer.layers.16.2.0.beta", "transformer.layers.16.2.1.weight", "transformer.layers.16.2.2.ds_conv.weight", "transformer.layers.16.2.4.gamma", "transformer.layers.16.2.4.beta", "transformer.layers.16.2.6.weight", "transformer.layers.17.0.q_scale", "transformer.layers.17.0.k_scale", "transformer.layers.17.0.norm.gamma", "transformer.layers.17.0.norm.beta", "transformer.layers.17.0.to_q.weight", "transformer.layers.17.0.to_kv.weight", "transformer.layers.17.0.to_out.0.weight", "transformer.layers.17.2.0.gamma", "transformer.layers.17.2.0.beta", "transformer.layers.17.2.1.weight", "transformer.layers.17.2.2.ds_conv.weight", "transformer.layers.17.2.4.gamma", "transformer.layers.17.2.4.beta", "transformer.layers.17.2.6.weight", "transformer.layers.18.0.q_scale", "transformer.layers.18.0.k_scale", "transformer.layers.18.0.norm.gamma", "transformer.layers.18.0.norm.beta", "transformer.layers.18.0.to_q.weight", "transformer.layers.18.0.to_kv.weight", "transformer.layers.18.0.to_out.0.weight", "transformer.layers.18.2.0.gamma", "transformer.layers.18.2.0.beta", "transformer.layers.18.2.1.weight", "transformer.layers.18.2.2.ds_conv.weight", "transformer.layers.18.2.4.gamma", "transformer.layers.18.2.4.beta", "transformer.layers.18.2.6.weight", "transformer.layers.19.0.q_scale", "transformer.layers.19.0.k_scale", "transformer.layers.19.0.norm.gamma", "transformer.layers.19.0.norm.beta", "transformer.layers.19.0.to_q.weight", "transformer.layers.19.0.to_kv.weight", "transformer.layers.19.0.to_out.0.weight", "transformer.layers.19.2.0.gamma", "transformer.layers.19.2.0.beta", "transformer.layers.19.2.1.weight", "transformer.layers.19.2.2.ds_conv.weight", "transformer.layers.19.2.4.gamma", "transformer.layers.19.2.4.beta", "transformer.layers.19.2.6.weight", "transformer.layers.20.0.q_scale", "transformer.layers.20.0.k_scale", "transformer.layers.20.0.norm.gamma", "transformer.layers.20.0.norm.beta", "transformer.layers.20.0.to_q.weight", "transformer.layers.20.0.to_kv.weight", "transformer.layers.20.0.to_out.0.weight", "transformer.layers.20.2.0.gamma", "transformer.layers.20.2.0.beta", "transformer.layers.20.2.1.weight", "transformer.layers.20.2.2.ds_conv.weight", "transformer.layers.20.2.4.gamma", "transformer.layers.20.2.4.beta", "transformer.layers.20.2.6.weight", "transformer.layers.21.0.q_scale", "transformer.layers.21.0.k_scale", "transformer.layers.21.0.norm.gamma", "transformer.layers.21.0.norm.beta", "transformer.layers.21.0.to_q.weight", "transformer.layers.21.0.to_kv.weight", "transformer.layers.21.0.to_out.0.weight", "transformer.layers.21.2.0.gamma", "transformer.layers.21.2.0.beta", "transformer.layers.21.2.1.weight", "transformer.layers.21.2.2.ds_conv.weight", "transformer.layers.21.2.4.gamma", "transformer.layers.21.2.4.beta", "transformer.layers.21.2.6.weight", "transformer.layers.22.0.q_scale", "transformer.layers.22.0.k_scale", "transformer.layers.22.0.norm.gamma", "transformer.layers.22.0.norm.beta", "transformer.layers.22.0.to_q.weight", "transformer.layers.22.0.to_kv.weight", "transformer.layers.22.0.to_out.0.weight", "transformer.layers.22.2.0.gamma", "transformer.layers.22.2.0.beta", "transformer.layers.22.2.1.weight", "transformer.layers.22.2.2.ds_conv.weight", "transformer.layers.22.2.4.gamma", "transformer.layers.22.2.4.beta", "transformer.layers.22.2.6.weight", "transformer.layers.23.0.q_scale", "transformer.layers.23.0.k_scale", "transformer.layers.23.0.norm.gamma", "transformer.layers.23.0.norm.beta", "transformer.layers.23.0.to_q.weight", "transformer.layers.23.0.to_kv.weight", "transformer.layers.23.0.to_out.0.weight", "transformer.layers.23.2.0.gamma", "transformer.layers.23.2.0.beta", "transformer.layers.23.2.1.weight", "transformer.layers.23.2.2.ds_conv.weight", "transformer.layers.23.2.4.gamma", "transformer.layers.23.2.4.beta", "transformer.layers.23.2.6.weight". 
	size mismatch for transformer.layers.0.0.to_q.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([512, 1024]).
	size mismatch for transformer.layers.0.0.to_out.0.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([1024, 512]).
	size mismatch for transformer.layers.1.0.to_q.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([512, 1024]).
	size mismatch for transformer.layers.1.0.to_out.0.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([1024, 512]).
	size mismatch for transformer.layers.2.0.to_q.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([512, 1024]).
	size mismatch for transformer.layers.2.0.to_out.0.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([1024, 512]).
	size mismatch for transformer.layers.3.0.to_q.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([512, 1024]).
	size mismatch for transformer.layers.3.0.to_out.0.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([1024, 512]).
	size mismatch for transformer.layers.4.0.to_q.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([512, 1024]).
	size mismatch for transformer.layers.4.0.to_out.0.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([1024, 512]).
	size mismatch for transformer.layers.5.0.to_q.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([512, 1024]).
	size mismatch for transformer.layers.5.0.to_out.0.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([1024, 512]).
	size mismatch for transformer.rel_pos_bias.net.0.0.weight: copying a param with shape torch.Size([512, 1]) from checkpoint, the shape in current model is torch.Size([1024, 1]).
	size mismatch for transformer.rel_pos_bias.net.0.0.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([1024]).
	size mismatch for transformer.rel_pos_bias.net.1.0.weight: copying a param with shape torch.Size([512, 512]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
	size mismatch for transformer.rel_pos_bias.net.1.0.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([1024]).
	size mismatch for transformer.rel_pos_bias.net.2.0.weight: copying a param with shape torch.Size([512, 512]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
	size mismatch for transformer.rel_pos_bias.net.2.0.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([1024]).
	size mismatch for transformer.rel_pos_bias.net.3.weight: copying a param with shape torch.Size([16, 512]) from checkpoint, the shape in current model is torch.Size([8

[dnns92]

I'm getting the same error. This is probably a scrollback due to the earler warning You are using a model of type mert_model to instantiate a model of type hubert. This is not supported for all configurations of models and can yield errors. Its probably related to the config yaml:

    "hubert_kmeans_cfg": {
        "model_name": "m-a-p/MERT-v0"

where indeed a hubert kmeans is built from mert weights. I am kind of lost how to fix this.

[northmachine]

I got the same problem. I change the --model_config to musiclm_large_small_context.json in the refered checkpoints on google drive and it works