yaydl is Yet Another Youtube Down Loader, written in Rust.

⌈⌋ ⎇ branch:  Yet Another Youtube (and more) Down Loader


Check-in [d1365e1f4d]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:yaydl 0.15.0: Supports setting the preferred Invidious instance via environment variables now. Closes #19.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk | release-0.15.0
Files: files | file ages | folders
SHA3-256: d1365e1f4d2bdfc84a3d9a5c17626dc1697d54838b84c53641fcc2c767ffdcc6
User & Date: Cthulhux 2024-08-04 13:05:16
Context
2024-08-09
10:27
yaydl 0.15.1: YouTube lost its proxy support in 0.13.0, it is back. Sorry! Also, as we download from Invidious anyway, Invidious is now an officially supported site. check-in: 02f7c03168 user: Cthulhux tags: trunk, release-0.15.1
2024-08-04
13:05
yaydl 0.15.0: Supports setting the preferred Invidious instance via environment variables now. Closes #19. check-in: d1365e1f4d user: Cthulhux tags: trunk, release-0.15.0
2024-07-27
21:10
yaydl 0.14.1: fixes youtube (closes #18), updates dependencies. check-in: 1688ab1dc1 user: Cthulhux tags: trunk, release-0.14.1
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to Cargo.lock.

︙︙
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910

name = "windows_x86_64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"

[[package]]
name = "yaydl"
version = "0.14.1"
dependencies = [
 "anyhow",
 "cienli",
 "clap",
 "env_proxy",
 "fantoccini",
 "indicatif",








|








1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910

name = "windows_x86_64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"

[[package]]
name = "yaydl"
version = "0.15.0"
dependencies = [
 "anyhow",
 "cienli",
 "clap",
 "env_proxy",
 "fantoccini",
 "indicatif",
︙︙

Changes to Cargo.toml. 

1
2
3
4
5
6
7
8
9
10
11

[package]
name = "yaydl"
description = "yet another youtube (and more) down loader"
version = "0.14.1"
authors = ["Cthulhux <git@tuxproject.de>"]
edition = "2021"
license = "CDDL-1.0"
repository = "https://code.rosaelefanten.org/yaydl"
categories = ["command-line-utilities"]
keywords = ["youtube", "downloading", "video"]





|








1
2
3
4
5
6
7
8
9
10
11

[package]
name = "yaydl"
description = "yet another youtube (and more) down loader"
version = "0.15.0"
authors = ["Cthulhux <git@tuxproject.de>"]
edition = "2021"
license = "CDDL-1.0"
repository = "https://code.rosaelefanten.org/yaydl"
categories = ["command-line-utilities"]
keywords = ["youtube", "downloading", "video"]
︙︙

Changes to README.md.

︙︙
72
73
74
75
76
77
78




79
80
81
82
83
84
85

For some video sites, `yaydl` needs to be able to parse a JavaScript on them. For this, it needs to be able to spawn a headless web browser. It requires Google Chrome, Microsoft Edge or Mozilla Firefox to be installed and running on your system.

1. Install and run [ChromeDriver](https://chromedriver.chromium.org) (if you use Chrome), the *typically* named [Microsoft Edge WebDriver](https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/) (if you use Edge) or [geckodriver](https://github.com/mozilla/geckodriver/releases) (if you use Firefox) for your platform.
2. Tell `yaydl` that you have a web driver running: `yaydl --webdriver <port> <video URL>`. (The drivers usually run on port 4444 or 9515, please consult their documentation if you are not sure.)
   *Hint:* If you need this feature regularly, you can also use the environment variable `YAYDL_WEBDRIVER_PORT` to set the port number for all further requests.
3. In theory, it should be possible to use more sites with `yaydl` now. :-)





# How to contribute code

1. Read and agree to the [Code of ~~Conduct~~ Merit](CODE_OF_CONDUCT.md).
2. Implicitly agree to the [LICENSE](LICENSE). Nobody reads those. I don't either.
3. Find out if anyone has filed a GitHub Issue or even sent a Pull Request yet. Act accordingly.
4. Send me a patch, either via e-mail (`yaydl at tuxproject dot de`), on the IRC or as a GitHub Pull Request. Note that GitHub only provides a mirror, so you'd double my work if you choose the latter. :-)









>
>
>
>








72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89

For some video sites, `yaydl` needs to be able to parse a JavaScript on them. For this, it needs to be able to spawn a headless web browser. It requires Google Chrome, Microsoft Edge or Mozilla Firefox to be installed and running on your system.

1. Install and run [ChromeDriver](https://chromedriver.chromium.org) (if you use Chrome), the *typically* named [Microsoft Edge WebDriver](https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/) (if you use Edge) or [geckodriver](https://github.com/mozilla/geckodriver/releases) (if you use Firefox) for your platform.
2. Tell `yaydl` that you have a web driver running: `yaydl --webdriver <port> <video URL>`. (The drivers usually run on port 4444 or 9515, please consult their documentation if you are not sure.)
   *Hint:* If you need this feature regularly, you can also use the environment variable `YAYDL_WEBDRIVER_PORT` to set the port number for all further requests.
3. In theory, it should be possible to use more sites with `yaydl` now. :-)

# How to set the default Invidious instance

For rather obvious reasons, including (but not limited to) the fact that Google tries to choke third-party clients, `yaydl` has been using Invidious as a wrapper since version 0.13.0. Now sometimes, the default instance hard-coded into `src/handlers/youtube.rs` *will* fail to work properly. You can use the environment variable `YAYDL_INVIDIOUS_INSTANCE` to change that: Just set it to the URI (including "https://") of [any other instance](https://docs.invidious.io/instances/).

# How to contribute code

1. Read and agree to the [Code of ~~Conduct~~ Merit](CODE_OF_CONDUCT.md).
2. Implicitly agree to the [LICENSE](LICENSE). Nobody reads those. I don't either.
3. Find out if anyone has filed a GitHub Issue or even sent a Pull Request yet. Act accordingly.
4. Send me a patch, either via e-mail (`yaydl at tuxproject dot de`), on the IRC or as a GitHub Pull Request. Note that GitHub only provides a mirror, so you'd double my work if you choose the latter. :-)
︙︙

Changes to src/handlers/youtube.rs.

︙︙
18
19
20
21
22
23
24

25
26
27
28
29


30









31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


use crate::definitions::SiteDefinition;
use crate::VIDEO;

use anyhow::Result;
use regex::Regex;
use scraper::{Html, Selector};


// Starting with yaydl 0.13.0, this handler uses Invidious instead
// of YouTube. In no way am I interested in playing cat and mouse
// against Google.



const INVIDIOUS_INSTANCE: &str = "https://invidious.materialio.us";










fn get_video_info(video: &mut VIDEO, url: &str) -> Result<Html> {
    if video.info.is_empty() {
        // We need to fetch the video information first.
        // It will contain the whole body for now.
        // Exchange the URL -> Invidious:
        let id_regex = Regex::new(r"(?:v=|\.be/|shorts/)(.*?)(&.*)*$").unwrap();
        let id = id_regex.captures(url).unwrap().get(1).unwrap().as_str();

        let invidious_url = format!("{}/watch?v={}", INVIDIOUS_INSTANCE, id);
        let local_url = invidious_url.to_owned();

        let req = ureq::get(&local_url).call()?;
        let body = req.into_string()?;
        video.info.push_str(body.as_str());
    }









>





>
>

>
>
>
>
>
>
>
>
>









|








18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


use crate::definitions::SiteDefinition;
use crate::VIDEO;

use anyhow::Result;
use regex::Regex;
use scraper::{Html, Selector};
use std::env;

// Starting with yaydl 0.13.0, this handler uses Invidious instead
// of YouTube. In no way am I interested in playing cat and mouse
// against Google.

// The environment variable YAYDL_INVIDIOUS_INSTANCE can be used to
// define the instance to use, otherwise, yaydl defaults to this:
const INVIDIOUS_INSTANCE: &str = "https://invidious.materialio.us";

fn get_invidious_instance() -> String {
    let invidious_env = env::var("YAYDL_INVIDIOUS_INSTANCE");
    if invidious_env.is_ok() {
        invidious_env.unwrap_or(INVIDIOUS_INSTANCE.to_string())
    } else {
        INVIDIOUS_INSTANCE.to_string()
    }
}

fn get_video_info(video: &mut VIDEO, url: &str) -> Result<Html> {
    if video.info.is_empty() {
        // We need to fetch the video information first.
        // It will contain the whole body for now.
        // Exchange the URL -> Invidious:
        let id_regex = Regex::new(r"(?:v=|\.be/|shorts/)(.*?)(&.*)*$").unwrap();
        let id = id_regex.captures(url).unwrap().get(1).unwrap().as_str();

        let invidious_url = format!("{}/watch?v={}", get_invidious_instance(), id);
        let local_url = invidious_url.to_owned();

        let req = ureq::get(&local_url).call()?;
        let body = req.into_string()?;
        video.info.push_str(body.as_str());
    }
︙︙
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122


                // Example: type="video/mp4; codecs=&quot;avc1.64001F, mp4a.40.2&quot;"
                // Fetch the video/mp4 substring:
                let mut mime_split = this_mimetype.split(";");
                video.mime = mime_split.next().unwrap().to_string();

                let relative_url = this_tag.value().attr("src").unwrap();
                url_to_choose = format!("{}{}", INVIDIOUS_INSTANCE, relative_url);

                // Only update last_vq if it's the best format yet.
                last_vq = String::from(this_vq);
            }
        }

        if url_to_choose.is_empty() {








|








120
121
122
123
124
125
126
127
128
129
130
131
132
133
134


                // Example: type="video/mp4; codecs=&quot;avc1.64001F, mp4a.40.2&quot;"
                // Fetch the video/mp4 substring:
                let mut mime_split = this_mimetype.split(";");
                video.mime = mime_split.next().unwrap().to_string();

                let relative_url = this_tag.value().attr("src").unwrap();
                url_to_choose = format!("{}{}", get_invidious_instance(), relative_url);

                // Only update last_vq if it's the best format yet.
                last_vq = String::from(this_vq);
            }
        }

        if url_to_choose.is_empty() {
︙︙