[download fail]: B站番剧页面的解析问题

[Nathaniel-Wu]

Website name: Bilibili

OS:: macOS

Video URL:: https://www.bilibili.com/bangumi/play/ss46089

Stack overflow

panic: runtime error: index out of range [1] with length 0

Additional context

B站番剧页面近期又进行了改动，以lux目前的request处理方式得到的html页面包含的番剧数据的json格式与以往发生了较大变化，只会包含当前ep链接的数据，或者是ss链接第一集的数据，而不像以前那样会包含当前番剧所有单集的数据。按照这种改动，其实我已经在我的fork里面简单修复了该问题：Nathaniel-Wu@6fa6685

没有直接发pull request是因为并没有完全解决问题，因为以ss链接下载时应该是要下载所有单集的，修复之后只能下第一集。调查之后发现在浏览器里拿到的html内嵌的番剧json格式相比之前并没有变化，而lux拿到的却不一样可能是跟header有关系，extractBangumi()拿到的html是通过下面这一行的到的，header为nil。

lux/extractors/bilibili/bilibili.go

Line 352 in 0883b0f

html, err := request.Get(url, referer, nil)

将这一行改成如下所示的方案之后，偶尔可以拿到跟浏览器相同的结果，但绝大多数时候会被给到简化的结果。

var headers map[string]string
if strings.Contains(url, "bangumi") {
	headers = map[string]string {
		/* header values copied from an actual browser */
	}
} else {
	headers = nil
}
html, err := request.Get(url, referer, headers)

暂时想不到B站还会通过哪些字段检查访问者种类了，因此发一个issue看有没有熟悉这一块的朋友帮个忙。

[Satoshi2310]

遇到了同样的问题，下载番剧时失败

[TongReG]

同样的问题，我试着加了不同种类的cookie但是许多剧集依然会有大概率请求不到，包括单集请求的时候也是。

PS C:\> .\lux -c cookies.txt -i -d ep787493

lux: version 0.24.1, A fast and simple video downloader.


URL:         https://www.bilibili.com/bangumi/play/ep787493
Method:      GET
Headers:     http.Header{
    "Accept":          {"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},
    "Accept-Charset":  {"UTF-8,*;q=0.5"},
    "Accept-Encoding": {"gzip,deflate,sdch"},
    "Accept-Language": {"en-US,en;q=0.8"},
    "Cookie":          {"SESSDATA=xxxxx; bili_jct=xxxxx;"},
    "Referer":         {"https://www.bilibili.com"},
    "User-Agent":      {"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36"},
}
Status Code: 200
panic: runtime error: index out of range [1] with length 0

goroutine 1 [running]:
github.com/iawia002/lux/extractors/bilibili.extractBangumi({0xc000154f00, 0x2e}, {0xc000688000, 0x3f1ce}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0x1, ...})
        /home/runner/work/lux/lux/extractors/bilibili/bilibili.go:99 +0xf12
github.com/iawia002/lux/extractors/bilibili.(*extractor).Extract(0xfeb2d1?, {0xc000154f00, 0x2e}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0x1, {0xc0001300a0, ...}, ...})
        /home/runner/work/lux/lux/extractors/bilibili/bilibili.go:362 +0x278
github.com/iawia002/lux/extractors.Extract({0xc0001300d8?, 0x0?}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0xa, {0xc0001300a0, 0xb}, ...})
        /home/runner/work/lux/lux/extractors/extractors.go:54 +0x351
github.com/iawia002/lux/app.download(0xc0003f2ac0, {0xc0001300d8, 0x8})
        /home/runner/work/lux/lux/app/app.go:275 +0x2e8
github.com/iawia002/lux/app.New.func1(0xc0003f2ac0)
        /home/runner/work/lux/lux/app/app.go:252 +0x691
github.com/urfave/cli/v2.(*App).RunContext(0xc000169860, {0x1c17800, 0x2598560}, {0xc000158000, 0x6, 0x8})
        /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.6.0/app.go:322 +0x928
github.com/urfave/cli/v2.(*App).Run(0x70?, {0xc000158000?, 0x2a61b64af68?, 0x10100000010?})
        /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.6.0/app.go:224 +0x2f
main.main()
        /home/runner/work/lux/lux/main.go:13 +0x39

[ericzhangdl]

遇到了同样的问题，不能下载番剧
panic: runtime error: index out of range [1] with length 0