SCREWY SCRIPTS
It's funny how I have ideas for posts that I note down but never
get around to writing, then someone else writes about the exact
same topic. I was going to write a retrospective post around the
19th of Oct., which is when I started writing this Phlog (though it
wasn't online yet at that stage), but it was going to be a long
post and I haven't wanted to spend that sort of time on it. But the
other day I saw that Undo has written a similar sort of thing:
gopher://sdf.org:70/0/users/undo/j058
But I've still been spending way too much time at the computer
lately so I won't start writing that now. Actually he talks about
not finishing projects. Of course in that regard I cheat a lot by
describing in the ideas section projects that I don't actually plan
to start in the first place, but one that has caught me lately is
that internet client thing. I haven't talked about it much since
describing the idea for the same reason, but I have set it up. Not
actually on the Atomic Pi because that only arrived a couple of
weeks ago so I set it up on a laptop instead. Except being left on
all day it only survived for two days before overheating and being
reduced to "system board error" BIOS beep codes, and my only other
x86_64 laptop was the one that I had just finished setting up to
replace my old 19 year old Thinkpad. So I kept using the old laptop
for a bit longer. Then the Atomic Pi arrived, but it only boots
UEFI, and I'd set up the system on the Micro SD card the old legacy
BIOS way (which is all I've ever known to be honest). So yesterday
I finally set up a copy of the card with a manual legacy/UEFI dual
boot (or whatever you call that) installation, and I'll try that
out later.
No I won't because I've started babbling and before long I'll have
been typing all afternoon. So shut up and get back on topic!
Alright grumpy, I guess I'm actually just here to share some shell
scripts. In spite of all my talk about being sick of messing about
at computers (actually mainly as something to do in breaks from
adding up expenses my the tax return, which I did finally get
submitted), when I saw the Unix 1-liner Five Fridays Funfair I
thought that was a great incentive to work on a bare-bones version
of a YouTube browser that I've been intending to write ever since
YouTube went Javascript-only a few months ago (I'm surprised nobody
else on Gopher has been complaining about that - I bet they would
if they'd been trying to browse it from a 19 year old Thinkpad with
a 1GHz PIII CPU though!). Plus I missed out on the last Gopher
event that I was going to participate in because that laptop is
about the only portable computer that I have with a working battery
and hefting it around the great outdoors for ROOPHLOC didn't seem
too appealing. I did think about trying to use one of the Nintendo
DS consoles that I've collected, but that was going to be quite
tricky. Actually as of this week I have a box full of quite recent
laptops including some little ones that would be quite well suited,
but that's because out of desperation I'm going to try getting into
the used laptop/parts business - which basically means becoming the
nerdy equivalent of a used car salesman. Well I might borrow one
later to, err, "test the keyboard", and make a late ROOPHLOC entry.
gopher://katolaz.net/1/U1FFF_2020/entries.gph
This was meant to be about scripts or something wasn't it? Oh yeah,
so I tried to make a one-liner YouTube browser that processes the
metadata output from youtube-dl, but it soon became obvious that it
was going to be way over the 128 byte length limit. But I kept
working on it anyway, adding features, and then I realised a way to
cheat. There's no limit on the length of the example, so just
define key parts of the script ahead of the one-liner that actually
uses them all. It was still a little tricky getting that "main"
function down to size, but I managed it with a byte to spare, and
thought it was a good joke. I'm not sure whether "The Unix Wizardry
Trustworthy Fellowship" really got the joke though, or didn't
approve, because they just left out the example entirely. So it's
pretty much useless as presented there. Hence I'll put the full
version of the multi-line one-liner here, and a more practical
version that runs as a script in the scripts section (later):
##################################################################
Youtube Browser in ELinks
ytb (){ [ "${1##*[!0-9]*}" ] && $YTBD --playlist-end $1 "$2"|$YTBE|$YTBS || $YTBD "$1"|$YTBE|$YTBS;elinks $YTBF;rm -f $YTBF; }
Usage:
ytb [MAX RESULTS] URL
Example:
$ YTBD='youtube-dl -e -g --get-thumbnail --get-description --get-duration'
$ YTBE='sed -e s/_webp// -e s/\.webp/.jpg/ -e s/.jpg\?.*$/.jpg/'
$ YTBS='csplit -z -f /tmp/yttmp -b %03d.txt - /^[0-9:]*[0-5][0-9]$/1 {*}'
$ YTBF='/tmp/yttmp*.txt'
$ ytb (){ [ "${1##*[!0-9]*}" ] && $YTBD --playlist-end $1 "$2"|$YTBE|$YTBS || $YTBD "$1"|$YTBE|$YTBS;elinks $YTBF;rm -f $YTBF;
}
$ ytb 5 https://www.youtube.com/user/FILMAUSTRALIA
Comment: In ELinks: "<" ">" to change tabs/results, "W" to wrap
text. Enable option "document.plain.display_links" in ELinks config.
for download/viewable video/image URLs.
"less" could be used instead of ELinks, with navigation between
results using the ":n" and ":p" commands.
Some X terminal programs can detect URLs by themselves.
##################################################################
It works with individual video links, playlists, user accounts, and
worked with youtube-dl's search function except that seemed to
break while I was actually working on this script (you used to be
able to do eg. "ytb ytsearch5:tildeverse" for five videos about the
tildeverse (maybe)), I just search DuckDuckGo with
"site:youtube.com" instead, or video.google.com does still work
without Javascript (though you can't disable the link redirects
like you can with DuckDuckGo). Add the "-f" option to the
youtube-dl command (YTBD) to specify the video format/quality you
want so it only gives you one video download link. The numbers
printed to the terminal are the lengths of the description files,
printed by Csplit, they're handy as a progress indicator.
I also submitted some other simple one-liners that I've mentioned
before.
And while we're here, of course this Internet Client thing has
involved all sorts of minor scripting. The latest one was
auto-login with telnet. First off: yes I want to use telnet - the
whole point of this thing is to avoid needing software that becomes
obsolete and SSH is an example of that which I've battled far too
much with already. Also I do want to use passwords even though it's
unencrypted because the most likely point of attack is from the
internet, so at my router (running OpenWRT), and I think a hacker
is more likely to try brute forcing usernames over telnet if they
get in than sniff the network traffic for the password when it's
sent. It's not much protection, against an attack that's unlikely
anyway, I know, but it makes me happy.
There are examples on the internet of using Expect for auto-login
with Telnet, so I did this at first and it worked fine. But I don't
have Expect installed everywhere, and it needs TCL, which needs a
list of other dependencies, and these are old systems without much
HDD space to waste. I tried using Chat, but I didn't get very far
with that. Then I found a suggestion on a forum that, over many
hours of messing with stty and other things, eventually evolved
into this:
##################################################################
#!/bin/sh
#Telnet auto-login
USER='[username]'
#Note: can't find a way to make letters "C" or "p" work in password.
PASSWORD='[password]'
HOST='[hostname / IP]'
localtty="`stty -g`"
stty raw -echo intr ^[ start ''
{ \
sleep 0.5; \
echo "$PASSWORD"; \
sleep 0.2; \
echo "stty \"$localtty\" &> /dev/null"; \
cat; \
} | telnet -l "$USER" "$HOST" 2>&1 | cat
##################################################################
The C and p thing has just got me stumped. It always reads C as
"interrupt" and p as "start" and I can't seem to change that no
matter what I do with stty. In the end I meekly changed the
password to avoid using those characters. Change the sleep commands
to "sleep 1" if your sleep only understands integers.
It does print something like: '$ stty
"2506:5:bf:8a3b:3:1c:7f:15:4:0:1:0:11:13:1a:0:12:f:17:16:0:0:2f:0:0:
0:0:0:0:0:0:0:0:0:0:0" &> /dev/null' on the terminal after log-in,
but I don't think it's worth adding extra complexity to hide that.
You can automatically run other commands the same way.
Here's the Expect version that doesn't take offence at any password
character that you use:
##################################################################
#!/usr/local/bin/expect -f
set timeout 20
set host "[hostname / IP]"
set user "[username]"
set password "[password]"
spawn telnet -l $user $host
expect "Password: "
send "$password\r"
interact
exit
##################################################################
I don't use Telnet for launching graphical programs (the Xcellent
part of the idea), but Rexec which simply includes a "-p
[password]" argument. Rexec (from GNU inetutils - which I'm getting
quite familiar with now) works very well for a program that nobody
seems to want to make distro packages for. The main problem I've
found is that I originally tested this with Dillo on my Pi Zero
(which some may remember I couldn't run Firefox on), assuming that
would be equivalent to Firefox as far a X goes. Nope, Firefox sends
a rendered image of the whole page view, which is basically like
uncompressed video when scrolling the page. It's mostly usuable
with 10/100 Ethernet (it probably helps that my screen resolutions
don't exceed 1024x768), but a bit too painful over my 802.11g WiFi,
so I've got Ethernet cables strewn about now. There's more to say
about that, but I'll leave it until I do a real follow-up post
about the Internet Client. It'll probably be titled something like
"Me and my Xcellent Ideas".
- The Free Thinker
Oh almost forgot, here's everything I found on the web relating to
YouTube browsing (I looked pretty hard for an existing solution
that I liked, but I'm picky):
* yt - Command-line browser written in Python
https://www.hecticgeek.com/yt-youtube-browser-ubuntu-linux/
* youtube-viewer - Perl script, Command line or GTK2/3, requires
API key from Google
https://github.com/trizen/youtube-viewer
http://www.webupd8.org/2015/02/youtube-viewer-complete-youtube-client.html
* straw-viewer - Perl script, Command line or GTK3, no API key
required
https://github.com/trizen/straw-viewer
* SMtube - Qt GUI, latest version probably uses youtube-dl back-end
http://www.smtube.org/
https://packages.debian.org/sid/smtube
-Some sort of interface here: http://www.tonvid.com/index.php
-Is it just a web-based proxy with a Qt browser reading from
localhost? Or from that website?
-Just a browser for the website, and the site itself is
closed-source:
https://sourceforge.net/p/smplayer/bugs/695/
The TONVID site might not be very reliable [indeed it hasn't been],
but seems to work OK as a stop-gap.
* Dillo-ydl - youtube-dl interface for Dillo written in Python
https://git.scuttlebot.io/%25oEzgqM849P03lR3tR7qwbjzw%2Fc%2BR%2Bv0oIoHyDgEDrbw%3D.sha256
* Invidious - web interface Youtube browser
https://github.com/iv-org/invidious
-Various online instances:
https://instances.invidio.us/
RSS:
This is _much_ faster than youtube-dl on my internet connection, so
should really try to use it instead.
https://www.youtube.com/feeds/videos.xml?user=FILMAUSTRALIA
- Unforunately doesn't work well with some RSS viewers. Nrss only
shows headings.
- can be sort-of viewed in links, but image links not displayed.
YouTube is now using WebP images, and these are supplied by
--get-thumbnails.
- However JPEGs still available as:
https://i.ytimg.com/vi/[VIDEO ID]/hqdefault.jpg
-So just use --get-id to grab the Video ID string and put it in.