I am trying to make an application that downloads something and puts it on an FTP server. There is not enough space to, even temporarily, put those files that are being downloaded on my computer, so I want them to download into the FTP server.
Specifically, I am using pytube to download a video using:
stream.download()
The .download() can hold a path variable, though I do not know how to use it to send the FTP server this way. I am looking for a way to open an FTP directory, so that I can use the path to fill in there.
In glitch you can download them to the /tmp folder which has a higher storage limit than the /app directory. What you want to do is download a chunk of video from youtube, then append to the file on the remote ftp server.
Here’s a stackoverflow post that describes in much more detail how to upload in chunks
I don’t know much about pytube but you can also get the raw url of a youtube video through libraries like pafy or youtube_dl. However if you want to stick with pytube I’ll give you this link to the documentation
It looks like you can override the on progress function to instead upload to the ftp server that way, or append to a queue of chunks to be consumed by the ftp server
Also another intresting solution but utilizing curl, if you can output the video source into stdout, you can pipe it into curl which can upload it into the FTP server.
Hey @SinglePaper! @javaarchive is right overriding on_progress and pushing onto a queue to send to the FTP server is probably the best option, sounded fun so I gave it a shot, below seems to work ok :).
import io
from threading import Thread
from pytube import YouTube
from ftplib import FTP, error_temp
from queue import Queue
class BytePipe:
def __init__(self, size: int):
self.q = Queue()
self.size = size
self.bytes_read = 0
def read(self, blocksize: int):
# if we have piped the whole file return False to ftplib
if self.bytes_read == self.size:
return False
chunk = self.q.get(block=True)
self.bytes_read += len(chunk)
return chunk
def write(self, chunk: bytes):
self.q.put(chunk, block=True)
if __name__ == "__main__":
ftp = FTP("127.0.0.1")
ftp.login(user="anon")
gangnam = YouTube('https://www.youtube.com/watch?v=9bZkp7q19f0')
stream = gangnam.streams.first()
pipe = BytePipe(stream.filesize)
# warning, hack begins here :)
# override the on_progress of Stream object and pipe chunks to ftp
stream.on_progress = lambda chunk, fh, br: pipe.write(chunk)
with io.BytesIO() as buffer: # we don't really need this buffer - we're piping straight to ftp - but create one anyway because pytube expects one
t = Thread(target=stream.stream_to_buffer, args=(buffer,))
t.start()
# retry on temporary errors
while True:
try:
# set the same block size that pytube defaults to
ftp.storbinary("STOR file.txt", pipe, blocksize=9437184)
break
except error_temp as e:
print("temp error, retrying...")
t.join()