Just repeat mutiple copy when opening tensor parallelism?

[huizhanyi]

I try the tensor parallelism by modifying offline_inference.py script

llm = LLM(model="opt-125m", gpu_memory_utilization=0.2, enforce_eager=True, tensor_parallel_size=2)

I print the hidden states in model script. I find the worker and driver worker use same input and output same result. So it seems tensor parallelism just executes multiple copys in each device.
Anything is wrong? Wish get some answers.

[huizhanyi]

it seems the weight are divided.

[jeejeelee]

I suggest you look at the LlamaMLP or OPTAttention code . Starting from there should help you understand TP quickly.