Question regarding "2nd Step - predictions"

[JavenLiPl]

Refer to the description: "For the inter prediction we need to send the motion vectors and the residual and the intra prediction we'll send the prediction direction and the residual as well.".

Why for inter prediction we need both montion vectors and residual?
Per previous chapters, it seems that using motion vector is more efficient using residual for inter prediction, and both are different ways to remove temporal redundancy. So why need to use both instead of just choosing one?

[leandromoreira]

Why for inter prediction we need both montion vectors and residual?

That's a good question, @JavenLiPl . I believe (strong emphasize in I believe) we need it, mainly because the motion prediction isn't flawless. It's not pixel-perfect; it usually uses a tree-like structure ranging from 4x4 to 64x64 to represent movements. Therefore, one might still need to send the compressed frame/residual.

I believe it increases the compression ratio of the residual compared to just a simple frames subtraction.

def diff_residual(frame1, frame2):
   return frame2 - frame1

def mv_residual(frame1, frame2):
   mv = predict(frame1, frame2)
   residual = frame2 - reconstruct(frame1, mv)
   return mv, residual

frame_diff = diff_residual(f1, f2)
mv, frame_mv_diff = mv_residual(f1, f2)

# I believe compression ratio of frame_mv_diff will be much higher
print(compression_ratio(frame_diff), compression_ratio(frame_mv_diff))

But I might be wrong. My (high-level) view is restricted by what I've read since I didn't create a functional video coded with inter-prediction.

I used mediainfo to check a h264 bistream and it seems that it carries the mv and the residual (compressed slice).

ref:

• https://www.vcodex.com/h264avc-inter-prediction/
• mediainfo --Details=1 video.mp4 | less

[ChristianFeldmann]

It is not a question of one or the other. The hybrid coding scheme relies on a combination of prediction and then lossy compression of the remaining residual. Lets back up one step:

Inter (motion compensation) and intra prediction are both just that ... predictions. They just have different spaces where the information is predicted from. For inter prediction, the prediction is done from previously encoded frames (which may be in the temporal past or future) and for intra prediction the prediction happens from information within the same frame (same temporal spot) that is currently being encoded/decoded (typically from neighboring block pixels).

Of course we are doing all of this to get a prediction that is as close to an original (the encoder input video) as possible. Because the better our prediction is, the less different information to the original we have to encode. This is the residual signal. It is the different between the original video and the prediction. No matter how the prediction is formed, we next get the residual and do a lossy compression of it.

So there is always a prediction happening for each block. Intra or inter. However, if the prediction is already close enough to the original, then the encoder may skip the encoding of a residual signal. This actually (depending on the settings and content) can happen a lot. Think of static background with no motion or very homogeneous areas.

I hope this helps. I will also shamelessly link my Youtube video here about Video Coding Basics.

[JavenLiPl]

Thanks a lot.

One other thing I am a little confused is that, if residual is defined as the difference between two adjancent frames, and the residual is sent as well along with the motion vector.

Then why we still need the motion vector? Per my understand, by using the residual, we are able to create the image flawless already.

Just new to the area, I might ask very fundamental and stupid question.

[leandromoreira]

@JavenLiPl I think the residual is acquired AFTER "applying" the motion vector predictions (so you have a lots of 0's easy to compress). No question is stupid :)