SWAN is currently approaching its fifth year pretty fast so it is maybe a good time to do some major improvements to our program. Here is an overview of what we currently have:
Use the every-24h data dump to get endorsement data from nations
A set of guide dispatches for SWAN and WA/endorsement-related matters
A dispatch that pings all non-WA nations that have been founded in the last 2 days and another one that pings all WA nations that haven’t endorsed the entirety of the Delegate+SPCG
5 awards: The endotarting award counts the number of times the endorse button has been clicked, the endorsement count award counts the average endorsement count a nation holds in a month, the number of nations endorsed award counts the average number of nations one endorsed in a month, the SWAN Knight award is awarded to people who have endorsed the SPCG and the Delegate at any point in the month, the WA award is awarded to people who have endorsed at least 20 nations and is in the WA at any point in the month.
Achievement
The delegate consistently has the 2nd highest endorsement count of GCR delegates. Percentage of WA nations that endorse the delegate is the highest or second highest of all GCRs. (~70-75%)
Percentage of nations in the WA on par with TNP (~10-15%)
Percentage of SWAN Knights among WA nations on par with TNP (~40-45%)
Regional culture heavily emphasizes the endorsement process
Drawbacks
Still low cross-endo rate (~9-13%) although it is a lot higher than in the past and is the 2nd best of all GCRs.
No formal position to govern/lead the program.
From these we can conclude that the model definitely does work as nearly all of our important metrics (the percentages) are only slightly behind or even better than TNP. We will never have our absolute numbers (e.g. delegate endorsement count) as high as TNP since they have a lot more nations than us but our engagement work on the SWAN Knight and delegate target is currently as good as them. What needs to be fixed in the low cross-endo rate.
The main culprit for the drawback is obviously the lack of a link to an endotarting tool on the dispatches. There are actually a couple of endotarting tools available now. TNP has one and we have one by ourselves, there are a couple of others that I don’t know. I am currently not clear on which is the best so I need some suggestions from you guys.
Improvements
Aside from fixing the drawbacks, we will have to add some additional features. The priority of our program now is to get as many nations to cross-endo on everyone in the region as possible. Putting an endotarting tool on the dispatches is the number one priority fix but we will also need to make some changes to our awards to incentivize this behavior first and foremost. I have a couple of suggestions:
Keep the endotarting, number of nations endorsed, swan knight award intact.
Remove the endorsement count award as this does nothing to incentivize cross-endorsing nations
All SWAN Knight awardees enter a card lottery. Remove the WA beginner award (You get one by being in the WA and gave out at least 20 endos) as you need to wait until the end of the month to give out this award, the incentive it provides is not enough for people to endorse more if we wait that long to give the award out.
Create a new award for people that endorse SWAN Knight nations. SWAN Knight nations are the people that have a high chance of sticking around long-term, so we may want to incentivize people to endorse these nations (just like TNP). This award works just like the endotarting one except easier to do since there are a lot less SWAN Knight nations. We can make this the endo award for beginners since it is easier to do.
Looking back the guide dispatches, I think they are still good enough and no modification to the content is needed outside of minor updates and syncing the style with the regular dispatches. Again, we need to add a link to the endotarting tool. One area with things to improve upon are the ping dispatches (Delegate+SPCG, WA). I think they are conceptually sound but better wordings or structure may be needed to encourage people to do what they need to do.
Another fun thing to add in is rewarding cards for award winners, perhaps guaranteed reward cards for champions and lottery for everyone else? We need more data from SPACE test run to decide on the details here.
One final thing, some people love statistics here so if you guys needs any more WA stats and rankings than what the dispatches provided then please post below.
Tl;dr
I need to know what is the best endotarting tool available for our program right now.
Some changes to the awards are needed as proposed above to incentivize cross-endorsing.
May want to improve the the ping dispatches.
May consider rewarding cards for award winners.
May add more statistics and rankings.
These are the ideas that I have now. If you guys have anything to say about this, please post.
Excited to see this! I’m not too familiar with Python (I recall that’s what this uses?) but would be happy to try and contribute if you’re interested
Some quick thoughts on the points you brought up:
Endotarting tool: the best one that I know of is Calref’s Tart. With respect to other tools, TNP’s hasn’t been working for me and I don’t think Glen is self-hosting Trove anymore so Archer hasn’t had a backend API to pull data from. Personally, I think it would be ideal if we could roll our own tool (again), especially because I’m not aware of any prominent working options that work with localid codes directly, which can be a pretty substantial speed-up for more serious endotarters.
Dispatches: I agree that they’re mostly fine, though I think they can be condensed a bit. I do wonder if the static content of dispatches could be replaced with a more interactive process involving NationStates <iframe>s on an external website — I can hack together a proof-of-concept if you’d like, and maybe it’ll be easier to evaluate its prospects from there?
Statistics: random thought, but a more interactive chart for the ‘Portrait of the South Pacific’ might be nice. I don’t know how easy converting GraphViz would be, but something like Vega that allows the data and visualization information to be serialized into an URL could be a low-overhead solution? I’ve never actually used it though, so I can’t vouch for it (and it’s also probably a rather long URL…)
I am not sure what half the technical stuff you said meant, but I can say, from experience (and a beginners POV), the easiest way to endotart (and thus get more people to) is to have it to where they can stay in one place and click endo buttons all on one tab.
What usually take several minutes (on a short day) or a couple of hours (for a new nation) would quickly become daunting to most nations. However, a centralized all-in-one spot (like I think TNP and Archer used) place turns that process into seconds and minutes, respectively.
I think it’s possible with some scripts to just click one button constantly to endorse nation, instead of opening individual nation pages, but that’s browser-specific to Chrome, iirc. Archer was definitely the most useful endotarting tool I’ve used, in that regard, since I’m on firefox and there’s no good add-on that has the same tool.
As far as I know, a single button to endorse a series of nations from a single tab requires either asking users to view the source code of a NationStates page (as Archer does) or to install a browser extension or desktop app. I don’t think those are huge hurdles though, and certainly speed up the process considerably since it eliminates the need to open links to nation pages, endorse those nations, and then close them.
Henn, I think you’re referencing Tart’s functionality? That button allows the user to open up a bunch of nation pages without changing tabs (on Chrome, at least) but the user still has to go to those tabs and endorse each nation. Arguably, that’s as convenient as it can get without the methods I mentioned above, but I think those tools aren’t really such a huge hurdle and are generally a worthwhile investment with the number of WA nations we have.
I’d also say that creating our own region-specific tool could help with things like always listing the Delegate and Coral Guard first, avoiding nations over the endorsement cap, and so on.
Archer’s method of viewing the source code was much more user-friendly than having to do everything via a new tab everytime, especially considering it’s just a webpage and not a separate application to install. I do wonder about the overhead needed to host another instance of Archer, but it might be slightly out of my reach to do myself.
Would there be a way to have a new nation get a list that requires them to endorse the Delegate + CG first, which would then “unlock” the ability for the rest of the endorsement list to populate?
I imagine it would be a bit technical, and it could have the unintended effect of lowering total endos, but I am throwing out ideas to see if they stick.
I’ve always said that we should have our own dedicated endotool and prominently advertise its use. For a while, I had been using TNP’s tool until it stopped working and, before that, a bot through Discord, but we’d definitely be better off with our own in-house tool. If we could have it hosted here on this forum, that’s brownie points right there.
Endotarting tool: the best one that I know of is Calref’s Tart. With respect to other tools, TNP’s hasn’t been working for me and I don’t think Glen is self-hosting Trove anymore so Archer hasn’t had a backend API to pull data from. Personally, I think it would be ideal if we could roll our own tool (again), especially because I’m not aware of any prominent working options that work with localid codes directly, which can be a pretty substantial speed-up for more serious endotarters.
So does this mean right now we should use Tart until we have our own solution? Archer should definitely be revived at some point though as our endotarting tool might be the most efficient with the use of localid (The last time in 2022 I try to use it this feature does not work though). May call up Glen on this.
Griffindor: Would there be a way to have a new nation get a list that requires them to endorse the Delegate + CG first, which would then “unlock” the ability for the rest of the endorsement list to populate? I imagine it would be a bit technical, and it could have the unintended effect of lowering total endos, but I am throwing out ideas to see if they stick.
It should be easy to list the delegate and SPCG nations first, however the dispatches already list the steps out so I don’t really think we need to go to that length. We never have an issue with delegate and SPCG endorsement, we have a low cross-endo rate issue so our focus now should be toward that instead of the delegate and SPCG.
Dispatches: I agree that they’re mostly fine, though I think they can be condensed a bit. I do wonder if the static content of dispatches could be replaced with a more interactive process involving NationStates <iframe>s on an external website — I can hack together a proof-of-concept if you’d like, and maybe it’ll be easier to evaluate its prospects from there?
I think we should just using regular dispatches. Using an external website sounds too extra. We ought to condense the dispatch material if needed. The process is not difficult to the point you need something interactive like the legislator form (And I personally think it is actually too extra as well).
Statistics: random thought, but a more interactive chart for the ‘Portrait of the South Pacific’ might be nice. I don’t know how easy converting GraphViz would be, but something like Vega that allows the data and visualization information to be serialized into an URL could be a low-overhead solution? I’ve never actually used it though, so I can’t vouch for it (and it’s also probably a rather long URL…)
I don’t know how good it would be at displaying 1000+ nodes and 100000+ edges though. GraphViz already throws warning at 1000 WA nation count. The UI could be very laggy. Some additional research on that front is needed. The source data is currently just an in-memory NetworkX graph object so converting it is very easy although I will use a database to store the graph data (so other services like an endotarting tool perhaps can use it) in the future.
Glen probably knows best, but I think Archer (the frontend web interface) works fine, but Glen’s not personally hosting Trove anymore so Archer doesn’t have an API to query which nations someone needs to endorse. The localid functionality should work fine, we just need an API to show which nations someone needs to endorse. If you want, I could probably throw together something rudimentary for that (even if just a bunch of static JSON files >_>) just as a basic proof-of-concept.
In the meantime, unless anyone knows of anything better, I think Tart’s our best option.
My main concern isn’t complexity, it’s UX. We’re actively directing users away from our dispatches to go to their settings page (if they need to set an email address) or the WA page (to apply), and then to their email, and we can’t really do better than just hoping they’ll remember to come back. It’s a simple process if you know it but confusing for a first-time user. I do agree that an external website is a bit extra, but it could theoretically solve some of these problems because users would be able to do all of these things without having to change tabs and we can even redirect users back to NationStates when they’re done. It’s maybe worth it, maybe not — I do think it might be exploring and seeing if we like it.
Actually, I was also about to say that the website content could be personalized to each user (such as if they’ve endorsed some but not all of the Coral Guard) but on second thought, we could probably just send personalized telegrams through the telegram API.
We can just put the link to the relevant pages on the dispatch + images if needed. That said, I will emphasize again that the WA and delegate engagement process is entirely alright in our region. It is as good as TNP (or even better depends on the season) so I think this is just unnecessary. The area we should focus on right now is increasing the cross-endo rate, and that means fixing dispatch content to better guide this process + have a better endotarting tool. The UX is easily the most horrible (with easy proofs) at the endotarting process so we must focus on that front. Once this stuff is addressed, we can do the website or telegram thing. Make the endotarting step the priority.
Archer definitely did not work for me with the localid thing. Tbf, I tested it on Firefox with a bunch of privacy options on so it might not work due to that. Will try it again at some point on the typical browser environment.
On the Trove thingy, we definitely need to have a central database for endorsement data in general since the SWAN script already needs endo data anyway. If Trove can deliver these data then we should start it up again not only for Archer but for the SWAN script. I plan to integrate near-real-time update to endorsement data using the API so this may make things better for both Archer and SWAN.
This is a slightly different version of Archer, which is why it’s not hosted at the tspforums.xyz URL.
This only works for nations in TSP.
This remains experimental, at least as of right now, so I wouldn’t get too dependent on it.
The data currently only updates as often as I manually run a command on my computer, so if the data you see is out of date, that’s probably why.
You might see yourself listed as a nation to endorse. I’ll try to get around to that soon.
For the technically inclined, the modified Archer code is here and the backend code is here.
Some more technical notes:
The ‘backend’ is really just a static website generated with Astro.
The core functionality is in src/pages/tart/[nation].ts where it essentially just downloads the daily dump, finds which nations each TSP resident hasn’t endorsed, and generates a static JSON endpoint with the list for each nation.
There’s (unfortunately) no central database or real-time data involved. A serverless/static hosting approach surpassed my initial expectations to be honest, even though I only really went with it because I didn’t have a personal server to run Trove on .
I think a Jamstack-style approach is more feasible than I expected. I’m assuming we already run SWAN daily (or something similar) to update the dispatches, so we can probably just generate and update the endotarting data for Archer daily in a similar way. That’s in contrast to Trove, which always had a server running that would make database queries to serve incoming API requests.
I think real-time data updates would be better handled on the client-side (i.e. by modifying the Archer JavaScript or making a new tool). If it was handled on the server, multiple users could get in the way of each other’s API rate limits since the actual requests to NationStates would be coming from the server’s IP address.
So, I tried the latest version of Archer, it gave me a bunch of nations to endorse (this was accurate because I purposely took a couple of weeks off to get below the endo cap).
I tried it again today* and also compared it to TART. TART gave me 23 nations to endorse (which sounds about right for two days of no endoing).
*Archer gave me the same list again because I am assuming that the next day’s dump was not put into the program (as Pronoun said would need to be done).
I’d say that Archer is indeed working (and the buttons allowed me to endorse much quicker than I otherwise could have).
I’ve just manually updated the site — feel free to try it again.
(On a technical note, I recall that Cloudflare recommends just using the headers they provide for reasonable caching, so things should just work, but I haven’t tested it yet.)
Automatic updates aren’t too hard but I do have to check on some free plan limits for the hosting provider I’m using. Part of me is also wondering if a separate codebase is worth it compared to folding this into the rest of SWAN, in line with Viet’s idea about having a central database.
An architecture I would do is to just have a database that stores the endorsement relations (i.e. the edges) with some basic nation data (e.g. the full names). Archer and SWAN will connect to this database to get endo data. A script will run every day or every few minutes to read the dump or the API happenings to update this database. Having a formal API like Trove to read the data from this database is nice if we can get it back up but it’s unnecessary. Direct connection to the DB is fine in our case.
We can package the endorsement database and its update script into a container image perhaps, Archer into another image, SWAN into its own image. Considering our forum is already packaged in a Docker image, this architecture looks very clean. I am not entirely clear on how Trove stores and updates its data and the overall setup of our server though, may ask Glen or Roavin for further answers.
I think real-time data updates would be better handled on the client-side
We definitely should store the already-endorsed nations on the client side for a temporary amount of time to prevent serving already-endorsed nations. A simple solution is to store the list of nations endorsed locally for 24 hours (or to be more precise, the time until the next dump update) and hide these nations from the server’s response on our page. This one is a little bit shaky though since if you accidentally un-endorsed someone you wouldn’t know it until the next day. This is fine if we want to set the thing up quick.
For now, reading API happenings to update the endorsement data on the server is the most accurate option. The endorsement happenings API refreshes every 28 seconds so there is no issue of over-stepping the rate limit here as we have no reason to monitor it in real time. No matter what we do, we cannot update endo data in real-time unless every time the user clicks the endorse button on our own site we send the update to the server.
Whose rate limit would that count towards? For example I’d rather not have a server polling the API every minute because I might want to use a triggering tool at update that would need to poll the API as fast as possible within rate limits, and having another tool running in a server could lead to exceeding the rate limit.
The server is the one that runs the update script on the central database, unless we are hosting a trigger bot on the server I see no reason why this can be a problem lol. The entire infrastructure will be hosted on our TSP server or something out-of-reach of any military stuff anyway. That said, if 28 secs isnt good we can just run it every minute or delay itself when the rate limit header is too high.
No, I know that the API rate limits by IP — but I think I misremembered something with the API rate limit rules. Sorry >_>
Archer is just a collection of static files; I suppose a Docker image would work but however it’s currently hosted works fine (aside from maybe a domain migration being ideal). That depends on having a client-facing API though, unless we want to give the database password away in the API request
To be honest, I’m really not knowledgeable about PHP, but it looks like Trove makes a MySql connection (here)? Someone can correct me if I’m wrong. Trove also seems to collect much more information than we need for SWAN purposes.
The Trove/Archer thingy can be considered one in my eye at this point lol. Anyway, SWAN will directly access the endo DB. We also let Trove use this DB.
