Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

finger.openttd.org #48

Closed
planetmaker opened this issue Jan 28, 2019 · 18 comments
Closed

finger.openttd.org #48

planetmaker opened this issue Jan 28, 2019 · 18 comments

Comments

@planetmaker
Copy link
Contributor

https://finger.openttd.org/versions.txt lists the 'latest' version for all products OpenTTD offers for download. It currently is not updated. That makes nightly servers virtually impossible.

The other stuff on https://finger.openttd.org/ might be nice to keep updated automatically, too

@TrueBrain
Copy link
Member

Finger always has been a bit of an issue. And especially with things like kubernetes (on which OpenTTD now runs), it is really difficult to do. I have been wondering about how to solve this.

For now, I think the easiest is to have a "finger" per release type. You can use this file for that:
https://openttd.ams3.digitaloceanspaces.com/openttd-nightlies/listing.txt

It will always show the latest release available; it is in CSV format, with date and name of the release. It bypasses the CDN, meaning you get the most up-to-date version (and not a potentially cached version).

The "branches" and "tags" on finger are completely superseded by GitHub. GitHub allows to fetch this information via their API, and is much more likely to be future-proof. So I would suggest we remove those completely from finger.

Not all downloads are migrated to the new CDN yet, which makes a good solution for finger a bit more difficult. But it was more important to get the new nightlies available than it was to get everything perfect. If there are suggestions what a better replacement for finger is, I would love to hear it. Is an URL like above sufficient? Or should we publish every nightly on GitHub? Or tag every nightly?

Feel free to leave any ideas!

@TrueBrain
Copy link
Member

Also, not less important, the URL to download binaries from has changed between the old system and new (from https://binaries.openttd.org to https://proxy.binaries.openttd.org). This URL might change again in the future. Not sure how to handle this cleanly. I think that is not really possible.

Biggest hurdle is that the ingame client uses https://binaries.openttd.org for ingame content, and it is unlikely we can deprecate that any time soon (because BaNaNaS is being rewritten, which tights into this). Or I have to find a way to automatically update the CDN with content of BaNaNaS.

Many possibilities, but most of all, tons of annoying issues :D

Either way, still thinking we should stop using "finger" in the current form, and go with the "listing.txt" per release type.

Still looking for additional input!

@claman
Copy link
Contributor

claman commented Feb 1, 2019

Given that listing.txt will be up-to-date, and that it already exists, that does seem like a good direction to work in.

What's the timeline like on the BaNaNas rewrite? Depending on the projected timing, it might just be easier to wait until that's done.

@Eddi-z
Copy link
Contributor

Eddi-z commented Feb 1, 2019

I don't think there is such a thing as a timeline around these parts...

@planetmaker
Copy link
Contributor Author

Ok, I don't mind so much when the download URL and the place to get version info changes. That's about a one-time update to scripts and fine.

However:
AccessDeniedopenttdtx00000000000002638e543-005c54aea9-d7f20f-ams3ad7f20f-ams3a-ams3
is my current result for proxy.binaries.openttd.org

listing.txt as given by you lists the revisions... but what URL could I give a script to get the URL from?

https://www.openttd.org/downloads/openttd-nightlies/20190118-master
https://www.openttd.org/downloads/openttd-nightlies/20190118-master-g07a40df9
both don't work. Nor any link I tried to construct with proxy.b.o.o or similar. What do I miss or not understand?

@TrueBrain
Copy link
Member

TrueBrain commented Feb 1, 2019

Navigate to https://www.openttd.org, hit the Download nightly button, find the file you are looking for, and there you go :)

https://proxy.binaries.openttd.org/openttd-nightlies/20190201-master-g5a5861f2/openttd-20190201-master-g5a5861f2-source.tar.xz

That is the URL where you can find the source, for example!

If you want the direct URL on the webpage:
https://www.openttd.org/downloads/openttd-nightlies/20190201-master-g5a5861f2.html

But I am guessing you want the download URL more than anything else.

Not sure why you are trying the 20190118 version; that is rather old :D

Not sure why I mentioned listing.txt; I think latest.txt is more useful for you:

https://openttd.ams3.digitaloceanspaces.com/openttd-nightlies/latest.txt

Hope it helps!

@TrueBrain
Copy link
Member

With the addition of releasees, it turns out that latest.txt is not sensible anymore. Is latest the testing or stable release? So I added to the listing.txt a new field name, which for example is always master for nightlies, but testing or stable for releases. And I removed latest.txt.

Please let me know if you are scripting things with this, and see other possibilities :)

https://openttd.ams3.digitaloceanspaces.com/openttd-nightlies/listing.txt
https://openttd.ams3.digitaloceanspaces.com/openttd-releases/listing.txt
https://openttd.ams3.digitaloceanspaces.com/openttd-pullrequests/listing.txt

@andreszs
Copy link

andreszs commented Feb 14, 2019

Hi, I reported the issue here and also suggested the implementation of meta tags.

The https://openttd.ams3.digitaloceanspaces.com/openttd-releases/listing.txt does not include stable releases, is this intentional? Where can we get info about latest stable release? Also, implementing a single XML file would be more clean than this multiple TXT files method...

Also, what's with the bizarre "openttd.ams3.digitaloceanspaces.com" URL? Where does it come from and how can we ensure this URL won't change in the near future? I think it would be more reliable to request them from the http://www.openttd.org server directly... Thanks.

@TrueBrain
Copy link
Member

Let me see if I get all the questions here ... (forgive me if I miss any).

Please create a new issue about 'meta tags', or even better, a Pull Request ;) I have yet to look into it, what the best practices are, etc. Tnx!

I will see what I can do about the URL itself. Currently it just points to the CDN, but as it doesn't support IPv6, having it wrapped in some openttd.org URL might be nice. You could use proxy.binaries.openttd.org, but those files are cached for at least a day on the edge. That is not ideal for these kind of files. So we will have to cook up some other solution. I will put that on my todo. Good suggestion.

Currently these URLs only list what has been published on the new infrastructure. So it is not listing any stable, as there are no stables on the new infrastructure. Currently, a combination of 'finger' and these URLs are needed for the complete overview (as where the files are located also differ between those two). Soon that will be fixed if we release our first stable (and finger can be deprecated completely). Or I might manually migrate an older stable to the new infrastructure. (depends a bit on the timeline for the next stable, I guess).

We won't be publishing any XML files anymore. XML is a very unfriendly format, both for machine and human. I considered YAML briefly for the listing.txt, but these text-files are easier to process by human and by script (both from reading as from writing point of view). I currently see no need to change the format (but, if presented a use case, this is something we can talk about). As it goes with these suggestions, as you present only a solution, I have no clue what the question is behind the solution, so I am a bit guessing here :) Feel free to show what you are trying to solve, and we can chat about it!

We won't be publishing a single file anymore with everything in there. Currently how 'finger' works is unmaintainable, especially as we want to allow building and publishing things like patchpacks etc. Currently I think it is sufficient to have these listing.txt files, and so far people I have talked with have had no issues. If you can explain the usecase you are facing making you ask that question, we can have a chat about it. (again here, you present a solution; I would love to understand the problem you try to solve, so we can think with you in what the best approach is).

There was a lot to unfold in your comment; I hope I have them all :) Lets please continue the talk, and see what we can do to make everyone happy here! This is all about collaboration :) Tnx!

@TrueBrain
Copy link
Member

Another suggestion that came up, that I am going to leave here, is to create a REST API which publishes this information. That would mean making this compatible etc might be a bit easier.

This just doesn't exist at all yet, and has to be created from scratch.

@andreszs
Copy link

Sorry for the delay, the idea behind the XML is to download a single file (faster and more reliable than multiple HTTP queries) in order to populate the info required for my OpenTTD Manager to show versions info, and allow direct download:

image

As you can see, all I need is version number, date, and download URL. I suggested XML because VB6 features very simple XML parser.

I strongly disagree with the comment about XML being an unfriendly format. Please check out the following sample XML that would be needed in this case, and let me know if you find it complicated. XML does become cumbersome and tedious when additional features such as namespaces or attributes are introduced, they could easily drive mad any developer / parser, but they are not required here. As a personal opinion, the guy who invented namespaces should be lapidated.

<?xml version="1.0" encoding="UTF-8"?>
<stable>
	<version>1.8.0</version>
	<date>2018-04-01 13:07 UTC</date>
	<download>
		<win9x>https://binaries.openttd.org/releases/1.8.0/openttd-1.8.0-windows-win9x.zip</win9x>
		<win32>https://binaries.openttd.org/releases/1.8.0/openttd-1.8.0-windows-win32.zip</win32>
		<win64>https://binaries.openttd.org/releases/1.8.0/openttd-1.8.0-windows-win64.zip</win64>
	</download>
</stable>
<testing>
	<version>1.9.0-beta2</version>
	<date>2019-02-09 21:41 UTC</date>
	<download>
		<win32>https://proxy.binaries.openttd.org/openttd-releases/1.9.0-beta2/openttd-1.9.0-beta2-windows-win32.zip</win32>
		<win64>https://proxy.binaries.openttd.org/openttd-releases/1.9.0-beta2/openttd-1.9.0-beta2-windows-win64.zip</win64>
	</download>
</testing>
<master>
	<version>20190220-master-g21f009dc78</version>
	<date>2019-02-20 19:01 UTC</date>
	<download>
		<win32>https://proxy.binaries.openttd.org/openttd-nightlies/20190220-master-g21f009dc78/openttd-20190220-master-g21f009dc78-windows-win32.zip</win32>
		<win64>https://proxy.binaries.openttd.org/openttd-nightlies/20190220-master-g21f009dc78/openttd-20190220-master-g21f009dc78-windows-win64.zip</win64>
	</download>
</master>

The ill-use of XML may have damaged its reputation, but the XML format at its core remains simple and elegant. The currently used text files could change and are not very useful, specially when their URL keeps changing and no one knows what info and in what order could they provide. XML avoids all these problems, and allows adding info without breaking current implementations.

The Previous Versions combo box is populated from some binaries URL, but this might be removed because that URL is not updated anymore either.

@TrueBrain TrueBrain added the help wanted Extra attention is needed label Mar 2, 2019
@Leffe108
Copy link

Leffe108 commented Mar 15, 2019

Just joining in a bit. I'm Zuu on the forums/irc and ran OpenTTD Auto Update for some years, but I do not plan to maintain it further. If I can say something, it is to if possible publish information such that it allow standard tools on platforms to use it. That being said I don't know any standardized tool on Windows to upload software using some sort of finger server.

For the big releases it may be possible to setup CI/CD to push to windows store. If it would be feasible to have a nightly channel there I don't know. If windows store is not possible, then maybe there is something like apt but similar for programs on windows we could work with that is open and reasonable to offer as either you use this or you download on your own or build it yourself.

@TrueBrain
Copy link
Member

TrueBrain commented Jan 26, 2020

Okay, after almost a year, I finally found time to tackle this issue. What I came up with will replace https://finger.openttd.org/ soon (tm).

https://cdn.openttd.org/ is the URL of our new CDN. It is fully browsable (and cached). What replaces finger are two files:

https://cdn.openttd.org/config.yaml: this tells how this CDN is build up. It tells which folders are indexed, and what is special about them, if anything. Most important for automation is the subfolders key, which can either be per-year or per-name. When looking in these folders you will figure out what they mean.

https://cdn.openttd.org/latest.yaml: this tells of each release what the latest version is. For example, in category "openttd" you will find "stable", "testing", "master", and several PRs. The version key tells you what the latest version is, and the folder where to find them. category + name is unique in this set.

In case you don't care about all releases, don't worry: there is a subset of this file in every subfolder. For example: https://cdn.openttd.org/openttd-nightlies/latest.yaml

In every release is, as has been for a long time, a manifest.yaml. This tells exactly what files are available, the size, the checksums, and more meta data. Example: https://cdn.openttd.org/openttd-releases/1.9.3/manifest.yaml

There is also a folders.yaml in folders which contains folders, in case you want to automate crawling it (saves you from parsing the index.html).

This is not yet in production, but will be soon. I need to automate a few more things; after that all traffic will be redirect here, and finger will be closed.

Found a bug or problem? Let me know.
If you have a use-case that can't be solved with this, also let me know. But this should be a clean replacement for finger (well, I switched from CSV to YAML, but that is about it :D)

@TrueBrain TrueBrain removed the help wanted Extra attention is needed label Jan 26, 2020
@TrueBrain
Copy link
Member

This is now deployed. See https://cdn.openttd.org/README.md for more information.

https://finger.openttd.org will stop working in a few days from now.

@duckfullstop
Copy link

https://finger.openttd.org will stop working in a few days from now.

Are you still planning on pulling this down, or maybe redirecting users to the new dataset (with instructions on how to access it, of course)?

@TrueBrain
Copy link
Member

I am planning to do so yes, just haven't gotten around to it. Will push the change tonight.

I don't really have a way to redirect people, as the formats are very different. As these URLs are mainly used for automation, there is not really a point. So the idea was to simply pull the DNS entry, and tools will simply stop working. I feel bad about this, but I also don't see any other way. Guess that is also the reason I haven't done this.

Just went through the stats. Turns out, only 2 users seem to still be using this URL. So I guess the damage is rather limited. Sadly, which-ever tool they run, it doesn't tell the User-Agent, or, pro-tip, put their email in the User-Agent, so I have no way of knowing which tool it is.

But .. such is life I guess .. after 15 years .. time to retire finger.openttd.org I guess. Anyone wants to say some words?

@andythenorth
Copy link
Contributor

"Finger, you served us well. You will be missed."

??

@Leffe108
Copy link

Leffe108 commented Mar 31, 2020

Thank you for the time with finger. It gave light and source of knowledge but nothing is eternal and now have time come to say good bye to finger.

~ also known as Zuu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants