Welcome to Day 24 of my Blogging challenge. For today’s Techie Thursday post, let us take a look at an issue that I faced while setting up one of the two *final* CDNs or Content delivery Networks for this blog. In a nutshell, I successfully uploaded thousands of pictures to a web folder, from which the images would be delivered to the website. In the process, the settings for the file changed, which could have caused a security issue. Four hours of effort wasted, but important lesson learnt. Check the damn permissions before uploading the file on the Internet!
This post has two parts. First is a condensed version, which is intended for most readers. it highlights what I set out to do, what the issue was, and how it was addressed. I also have a longer “deep dive” version which is for a more technical audience.
In this post
1. Background : setting up a CDN for Images
3. How the Issue was Identified
4. Advantages of URL Based Transformations
5. Deep Dive: What caused the Error
7. Wrapping Up
The Short Version: Goof Up and Recovery
Background – setting up a Content Delivery Network for Images
You might find a few posts about CDN and the issues I have faced with images on this site. To be fair, many of them are issues of my own making. To address this, for the past few days I have worked very hard to bring all images together. And to give them some sense of structure and sanity.
Last night, the focus was consolidating images and uploading the to server for this blog. My goal was to to upload ~ 5,000 images on to a web folder and serve them through Gumlet CDN.In the process, the permissions for some of the images got changed. This could have potentially caused a security issue.
This is set up on a shared hosting site as a subdomain. This folder is linked to for my blog through a service called Gumlet.
Errors Galore: All went well, until permissions for image changed
Everything was set: over 5,000 images were uploaded overnight on to that web folder. These images were saved in different folders that were mapped to categories on this blog – such as Travel, Writing, etc. I felt proud of my achievement!
But the trouble began when I tried to access the images from the web browser, I began to get the dreaded 404 error (file not found).
I checked in the web folder; the images were there al right. And I had typed the path correctly. But then I noticed something odd.
Lo and behold, “hundreds” of images have wrong permissions (755) and some had the “executable” tag.
Something terrible had gone wrong. Folders are supposed to have permission of 755 and files should have permission of 644 (in plain simple terms, files should only be readable by the visitor). here I was, sitting on hundreds of files, with wrong (and dangerous) permissions. To understand more about permissions for files and folders and their significance, you can read here (Wikipedia) and here (Linode).
How the issue with image permissions was identified
I had uploaded the images on the server, but could not access them from a webpage. I have created a test site before migrating the setup to this blog. Let us call that site as myblog.tld.
Suppose, you visit my blog at myblog.tld, and click on any post, such as myblog.tld/mypost.html.Let us consider there are three images on that web page, called image1.jpg, image2.jpg and image3.jpg
My aim was to use Gumlet for serving images via their Content Delivery Network on to this blog. But before that, I had to ensure that the images were accessible via web browser. How it should have worked (as intended) is shown in the image below.
Below image should load from CDN once set up.
https://avblog.gumlet.io/Site/Banners/AmarBlog-HeaaderBanner1-Jan22-min.jpeg
When I checked my local folder on my Thinkpad, some images had wrong permissions, some were ‘executable’. In plain and simple terms, by using these images, anybody accessing the website could read, write and perform operations on the image files. This is a big no-no and a potential security risk. I should have checked the permissions before uploading to the server. To be fair, I had never experienced such an issue before, therefore it was missed out as a part of “sanity checks.” Ignorance is certainly not an excuse.
Corrective Action and Lesson Learnt
I deleted All images from Shared hosting server, am re-setting permissions locally, and will re-upload. This is a huge pain in the rear end. Re-uploading these files of about 8 GB in size will take me 3 or 4 hours. But this is hopefully a one time task. However, it is a must do.
Importance of checking permissions
Check the file permissions in the future, silly! That note was for me, and not intended for you, the reader 🙂
The Long version of this post
The Chaos with Images
When I began consolidating images, I noticed the following issues:
The images were all over the place- duplicates, multiple formats, multiple versions of the same image (i.e. different resolutions). I had over 75,000 images totaling over 400 Gigabytes. These are the perils of one or more of the following:
a. Taking images using multiple devices over years (digital camera, phones that use jpg or HEIC formats, etc. and screenshots from mobile phones and computers)
b. Downloading from various sources (own images plus public domain/ creative commons sources)
c. Storing them on different servers over the years (for example- using cloud storage like Google Drive or Photos, SaaS apps like Droplr, and storage servers like Hosthatch VPS)
d. The Images are in different formats such as jpg, png, heic, tiff, WebP, avif, svg …
These images were taken over 25 years, across multiple devices, stored on different servers, saved in different format. It was an absolute nightmare to consolidate them.
Bringing some sanity
The images are now consolidated into three main categories:
– Lower resolution images for sharing on websites and social media (including this blog)
– Higher resolution images of friends, family and other important persons and events.
– Images of my wife’s paintings (both low and high resolution)
The images are stored in three main formats: jpg, png or WebP. The WebP images are converted from png and jpg.
Using image CDN – Primary and backup service
I have written several times about using a CDN and its advantages. It works well when it does. In my case, sometimes it did not. Mostly because of my own mistakes.
At the risk of repeating myself, find the summary as follows. Suppose, you visit my blog at myblog.tld, and click on any post. In this case, the link is myblog.tld/mypost.html
Le ut suppose there are 3 images on that web page, each could be different format or type and have different dimensions (width x height) as follows:
image1.jpg say it is 1000×750 px (Landscape format)
image2.png it is 2000x 2500 px (irregular format)
image3.webp it is 600 x 1200 px (tall format, like in Instagram)
I would prefer the images to load consistently and in similar efficiency on different screen sizes, whether it is a desktop of 27 inches screen or a 6 inch mobile screen. The same images can be used on multiple pages, so if I can set the parameters one time (such as width, resolution, quality, etc.), I need not replicate them every time. Let me explain:
In typical HTML, I would have to hard code the size, adding parameters like width and height. In my case, I would want image1 to be say 50 percent smaller, image 2 to be 80 percent smaller. Image3 can remain as is- since it is WebP format, by design it will have smaller size in kilobyte terms.
<img src="/path/to/image1.jpg" alt="description of image1.jpg" width="500" height="375" />c
<img src="/path/to/image1.png" alt="description of image1.jpg" width="400" height="500" />
I can of course also type
<img src="/path/to/image1.jpg" alt="description of image1.jpg" width="50%" height="50%" />
or variations of the above. The images are now quite close in size, compared to before. This will have to be repeated on every page where the above image(s) will be used.I suppose the same can be done using CSS, but we will skip that for now- also because I don’t know how to do that… yet.
CDN with URL based transformations
Services like Gumlet and Cloudinary (and Publitio, and BunnyCDN…) allow you to load parameters like width, height, quality, etc… in the URL itself. You can upload and store the images on Cloudinary, BunnyCDN, Publitio servers… but in Gumlet you have to “link” them to a storage. In my case, I was using a web folder on shared hosting service.
Let us call that folder as /path/to/images on a subdomain images.myblog.tld.So if the images are stored on images.myblog.tld/path/to/images, we ‘link’ it to Gumlet using CNAME. Thus, cdn.myblog.tld is mapped to images.myblog.tld/path/to/images
Of course the corresponding DNS entries will have to be made with your domain registrar or Cloudflare, etc… explaining that is beyond the scope of this post, but you can read this fine post that explains how.
Advantages of using URL based transformation for Images
Instead of typing a long address or code like<img src="https://images.myblog.tld/path/to/images/image1.jpg" alt="description of image1.jpg" width="500" height="375" />
We can simply type
<img src="https://cdn.myblog.tld/image1.jpg?w=500&h=375&q=80%" alt="description of image1.jpg" />
Gumlet actually picks up the image from images.myblog.tld and serves them from cdn.myblog.tld.
Typically, this happens in the following order:
a. Connect to images of different format from images.myblog.tld/path/to/images/
Gumlet actually picks up the image from images.myblog.tld and serves them from cdn.myblog.tld. Typically, this happens in the following order:
i. Pick up images of different format from images.myblog.tld/path/to/images/
ii. Transform and cache them on Gumlet’s server
iii. Serve them on myblog.tld based on the requirements.
The advantage we get is
It gives a cleaner URL structure. This is good for search engine optimization and google results.
The image parameters can be changed for ALL images before they are served by Gumlet. So if I have a few thousand images (as is the case with this blog) then I need not pass the parameters for each and every image.
i.e I can set all images to have a quality of 90 percent, width of say 500 on Gumlet Dashboard). The images will be served as something like
cdn.myblog.tld/image1.jpg?w=500&q=90cdn.myblog.tld/image2.png?w=500&q=90cdn.myblog.tld/image3.webp?w=500&q=90
Essentially, the CDN (Gumlet) does the heavy lifting. What a huge time saver for both the web host and the website visitor!!!
Deep Dive: Root cause of the issue with permissions
There are two approaches to identifying an issue: Find the root cause, fix it and re-do the work. Second approach is literally “Shoot first, ask questions later”. For the former, I will have to:
- Check the logs in case there has been any intrusion or any suspicious activity, or it was human error.
- Double check the permissions folder by folder
- Re upload the images, and
- And re-set the image parameters. on Gumlet.
Fortunately some of it can be script based or automated, by still a pain in the rear end. The second approach is what I followed, considering the need of the hour. More on that in a bit. To help in the deep dive approach, user “Tim” on LowEndSpirit Forum helped me understand the situation (and provided a possible solution) by asking the following questions:
Blame it on the shared hosting control panel for accepting executable image files?
Blame it on the webserver for serving them (even to Gumlet)? Blame it on Gumlet?
Blame it on the shared hosting control panel for not allowing you to change the permissions?
How did all the images get the 7xx permissions in the first place?
I started with the hypothesis that it was human error (i.e. my mistake) for all questions. Or incorrect settings configured on my side.
Coming to think of it, the images with 755 permissions were not accessible via web browser. This was true, whether I used the gumlet address or tried to access directly by following the link for the image. In fact, this was the very reason why I started looking into the server logs and folder permissions. This search brought to light the issue with the permissions. The good news is that server(s) did not allow the images to load. The bad news is that the server allowed me to upload the images.
The images were ‘tar.gz’d and uploded using Filezilla. Maybe the server did not check permissions for the content of the archives. Upon extracting the archives, server did keep the permissions as 755.
That is a question probably for the webhost to resolve.
Fixing the Folder and Image Permissions
I started manually changing the permissions at directory level (not all files individually, but by selecting all files in the folder). This was slow process but gave a good visual sanity check. I use Manjaro’s XFCE edition, where the file manager is Thunar. Once again, user “Tim” on LowEndSpirit Forums helped out. He suggested that I use the below command to fix the permissions for all image files.
$ bash:~$ chmod --recursive 644 *jpg *jpeg *png
Alternative Command for changing permissions for files recursively
This command actually worked better, and voila! In a minute the permissions were fixed. Source: https://phoenixnap.com/kb/chmod-recursive
From the site, I used the following commands
sudo find Example -type d -exec chmod 755 {} ;
sudo find Example -type f -exec chmod 644 {} ;
This command worked like a charm!
Sample Output for the entire images folder
$ from ls -la -R Blog-Images-Feb22/Travel/India/Lepakshi-Dec2017 total 4844 drwxr-xr-x 2 amar amar 4096 Feb 23 03:53 . drwxr-xr-x 20 amar amar 4096 Feb 23 04:52 … -rw-r–r-- 1 amar amar 164087 Jan 12 07:29 IMG_0124_723429263.JPG -rw-r–r-- 1 amar amar 151680 Jan 12 07:29 IMG_0127_1644419028.JPG -rw-r–r-- 1 amar amar 168266 Jan 12 07:29 IMG_0128_504049756.JPG -rw-r–r-- 1 amar amar 223580 Jan 12 07:29 IMG_0129_964472840.JPG -rw-r–r-- 1 amar amar 229864 Jan 12 07:29 IMG_0131_1451517638.JPG -rw-r–r-- 1 amar amar 285952 Jan 12 07:29 IMG_0132_1214657274.JPG -rw-r–r-- 1 amar amar 188639 Jan 12 07:29 IMG_0134_1956957717.JPG -rw-r–r-- 1 amar amar 210415 Jan 12 07:29 IMG_0135_65944692.JPG -rw-r–r-- 1 amar amar 241848 Jan 12 07:29 IMG_0136_1495422938.JPG -rw-r–r-- 1 amar amar 935987 Jan 12 07:30 IMG_20171230_123849.JPG -rw-r–r-- 1 amar amar 915493 Jan 12 07:30 IMG_20171230_124028.JPG -rw-r–r-- 1 amar amar 846107 Jan 12 07:30 IMG_20171230_124329.JPG -rw-r–r-- 1 amar amar 352738 Jan 12 07:30 IMG_20171230_125155.JPG
Wrapping Things Up
This experience set me back by a few days with my goal of fixing my blog. But this was an experience that taught me a lot.
This post was written as part of #bloggingchallenge for February 2022 under the theme Techie Thursday. You can follow the progress on the Bogging Tracker Page.
Wordcount:2,100 Time taken for writing, editing, publishing: 2.5 hours.
This post was updated on 2022-06-04
