Archiving Websites
WGET and TREE
wget is a program that 'get's content from web servers. With 'wget https://yoururl.com' command we can 'download' any website. We can also download the images and files displayed on the website but link to other websites(-p or --page-requisites), convert all links to link to the local folder(-k or --convert-links), run multiple commands of wget on 1 line(-e or --execute) and has a mirror option(-m or --mirror). Using all of these commands we can get a website in its entirety locally. More and better explanation here.
wget -mpEK http://mywebsite.home.xs4all.nl/
Tree is also a program that creates 'tree's of folders and the files inside. With tree you can write a short command that decides on the level of the crawl(-L 1 : take the files and folders inside this folder, -L 2 take the files and folders inside this folder and if there are folders go inside them and catch those as well, etc.). Using tree we can also make html files that include the file tree we created. Without the html command the file tree will be displayed in the terminal. Explanations here.
tree -L 1 -H ./ > my.html
To archive websites, I wrote a short python script that saves websites using a wget and a tree command. Here is my archive.py:
import subprocess
import sys
#my function for wget, runs wget -mpek on the terminal and asks kindly to paste the website link
def wget():
print("paste website link(CTL + C to quit)")
reply = input()
#take the reply and use it in the wget command
subprocess.run(["wget", "-mpEK", reply])
wget()
#loop this function until CTL + C interrupts, I wanted to make a
#if reply is quit quit kind of function but
#in the grand scheme of things, it does not matter
while True:
wget()
#if you command this out, it might work but I didn't test it, I always use
#tree in terminal because i find this command very confusing...
## subprocess.run(["tree", "-L","1","./",">","my.html"])