0

Looking for some assistance or ideas on how to grab the latest version of a Tomcat release from their website or other website.

I came across these Stack Overflow/Stack Exchange links that were relevant below:

..But unfortunately both led to some dead-ends. It did give me some ideas though.

I have some installation scripts where the download URL is predictable based on the version number of the software, so if there's a consistent way of scraping for the latest version of the software I simply add a variable into the script for it to curl / grep for that version on the text and use that variable for download links, untaring, moving, etc.

Hoping to do something with Tomcat in the same vein. A installation script I won't need to keep updating (unless they change their site).

Any sort of thoughts, ideas, are appreciated.

I've also looked at http://tomcat.apache.org/whichversion.html which lists the "Latest Released Version" and this seems like maybe the best, more reliable location to grab the version, but I had trouble getting that with Curl / Grep because of the table structure. I am only just scratching the surface with learning those commands, so maybe someone more well versed could get that working, otherwise I'm definitely open to other thoughts!

curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"

It seems to work but I am not sure if I am entirely happy with it. An additional use-case that I thought of, is it would be nice to grab the latest version of each major version instead of just the absolute latest.

Do you have suggestions on how you would parse the highest number with the link you provided, and/or how to start in one of those directories and parse the highest there? That way for example I could create a latest version install script for Tomcat 7, 8, 9, etc.

Thank you!! (Like I said I am just getting started, so please my apologies with the lack of knowledge but I do appreciate digging in and understanding how the commands work or reached their conclusion)

Keegan Jacobson
  • 1
  • 2

2 Answers2

0

I've created a couple crude methods of grabbing the latest major versions of releases, as well as absolute latest.

# Grab Latest Tomcat 7
curl -i https://www-us.apache.org/dist/tomcat/tomcat-7/ | grep -Po '(?<=(<a href="v)).*(?=/">v)'

# Grab Latest Tomcat 8
curl -i https://www-us.apache.org/dist/tomcat/tomcat-8/ | grep -Po '(?<=(<a href="v)).*(?=/">v)'

# Grab Latest Tomcat 9
curl -i https://www-us.apache.org/dist/tomcat/tomcat-9/ | grep -Po '(?<=(<a href="v)).*(?=/">v)'

# Grab Latest Tomcat
curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"

It's a bit of a hack where I'm looking for text in-between two values on a web page to grab the version number. It's crude and I'm sure there's a more elegant way to grab this, but this what I was able to figure out with my skillset! Maybe it can help someone in the future, but if anyone has better ideas I'm all ears.

Keegan Jacobson
  • 1
  • 2
0

In general, I would not recommend scraping version numbers and download URLs from websites. You will end up having to modify your scripts when they inevitably end up breaking — I have learned this the hard way time and time again.

For example, take the following script I wrote in 2016 to scrape the latest Tomcat version:

curl https://tomcat.apache.org/whichversion.html | awk -n '
/class="detail-table"/ {
    phase = 1
}
phase == 1 && /<th>/ {
    target_column++
    if (/Actual release revision/) phase = 2
}
phase == 2 && /<tr>/ {
    current_column = 0
}
phase == 2 && /<td>/ {
    current_column++
}
phase == 2 && current_column == target_column && !/alpha|beta|superceded/ {
    print substr($0, match($0, /[0-9]+\.[0-9]+\.[0-9]+/), RLENGTH)
    exit
}'

Unfortunately, that script no longer works, and I do not know when it broke. No amount of fancy scripting will save you from unexpected structural changes to a website you are scraping.

Sure, maybe a little bit of finagling will fix this one particular script, but if you try to download all your dependencies this way, you'll be playing catch-up till the end of time. Aim for a solution that you only need to code once.

EDIT: There's a good chance that I've been successfully nerd-sniped by @Bill_Stewart, but a working solution today (October 18th, 2024) is to simply replace /Actual release revision/ with /Latest Released Version/ in the above awk script. My original sentiments still stand and I would never use this in production unless I literally had no other choice.

eeowaa
  • 105
  • 2