Dev Tracker

At approximately 14:00 Server Time on Friday, March 31 the engineering team was alerted to an individual probing for XSS (Cross-Site Scripting) weaknesses by pasting HTML script tags in various text fields on the site, some player-facing and some only visible to staff. Out of an abundance of caution we put the site into maintenance so we could evaluate the situation.

During the maintenance period, we reviewed the attacker's activity, focusing in particular on what fields the attacker typed script tags into. Then, we reviewed what pages (player-facing and staff-facing) display the contents of those text fields, whether those pages are properly escaping the text in question to prevent XSS vulnerabilities, and whether anyone viewed those pages while they potentially contained text content from this attacker.

Because it is part of our development practices to evaluate all new and revamped/refactored features for XSS vulnerabilities, we could not find issues in player-facing areas such as Forums, Private messages, Clan Info, or Dragon Biographies, or our primary staff-facing tools. We did identify and correct issues in some of our rarely used staff-facing tools that were not used while this attacker was active. Additionally, we identified some extremely old code that had potential issues and corrected them, but these areas are not areas where content entered by the attacker could have been viewed by another player or staff.

While XSS has a broad scope, one of the major concerns in any XSS attack is the potential that session cookies—tokens stored in your browser that "prove" that you are a particular logged-in user—could be stolen by an attacker, allowing them to essentially be "logged in" as you without knowing your password. Our session cookies are all set to use the "HttpOnly" flag which means that scripts categorically cannot access them. Barring vulnerabilities in browsers themselves, we do not believe session cookies were immediately at risk. Nonetheless, we revoked all outstanding staff and volunteer moderator sessions early on in our investigation as a safety measure.

Again, while we do not believe player sessions were at risk for the above reasons, if you wish to revoke all outstanding sessions for your account, including those for browsers/devices other than the one you are currently interacting with the site with, you can change your password either via Account Settings or the Forgot Password feature.

In addition to reviewing our existing code, we also looked into ways to detect and block this sort of behavior proactively. We have made some initial changes in that area and we are going to continue to improve our security posture by adding additional layers of protection, detection and alerting. Please bear with us as some of these changes may introduce minor bugs while we fine-tune things.

In summary, at present we believe this individual was at the stage of probing for vulnerabilities, and we used the maintenance period to review and strengthen our protections against this sort of attack. We do not have reason to believe the attacker accomplished anything of major concern at this point.

If you believe you have found a vulnerability of any kind anywhere on Flight Rising, please disclose it to us privately using Contact Us right away.

Thank you for your patience and understanding.

@Fennelroot Yes, the elemental festival currency items now drop in all elements! The chests still drop in the festival's specific element.

@Fruitgummysnake We granted an additional 30 (15+15) gathering turns, rather than setting them to 45, so depending on your well-fed bonus and your element's dominance position you could end up with between 40 and 48 turns. Hope that helps!

Hello @DiamondDustSky -- We checked our results and it looks like all of the turns were correctly used, although I'm sorry you could not see what you received! I suspect that what happened is that the new Gathering refactor page allows you to double click the button, which causes 2 turns to be used but only 1 result to be displayed. So if you are rapidly clicking the button turns may appear to drop fast and you will not see the results properly.

We will look into correcting this so you can't accidentally skip seeing the results of a turn.

Hello all, we have checked and confirmed that the festival chests are correctly configured and are dropping in Lightning Digging, there is just an issue with the Game Database that prevents them from displaying because they are not permanent Gathering drops.

@TheHunterZ Yes, we awarded +15 once when we discovered Charged Sprockets weren't dropping, then another +15 when we discovered an issue had caused the drop rate to be set incorrectly.

@Peculiarity Looking at our records, it looks like you used your first 15 turns of the day before Thundercrack began, several hours ago, and then 15 (the first awarded 15) after it began. Sorry for the confusion!

EDIT: For clarity's sake, since I started typing the original post, the second 15 was also used, so our records show a total of 45 turns used since rollover.

This issue should be fixed now! Thank you for the report.

We have updated the skin chests to drop in Digging in order to match the description given in the announcement post for the festival. We apologize for the confusion!

@RedWillia While the old Gathering implementation displayed it this way as well, you are correct that it is inconsistent with other features, I will bring it up with the wider team!

We were able to capture some data that points to an intermittent packet loss issue and have forwarded it to our hosting provider. At this point we are waiting to see if our hosting provider is able to make a fix based on what we have provided them. Thank you for your patience, everyone!

As an update, we have deployed a fix that may alleviate the issue somewhat. We have not reproduced the exact situation as the reports in this thread, but we have a theory as to why it might occur, so we are trying a simple, relatively low risk fix first.

Hello all,

We are investigating the possibility that this issue is related to the other site lag detailed here. It may be that this bug has existed for a very long time but only occurs under very specific circumstances where randomly long load times cause a race condition between one side loading and the other side loading. If that is the case it may be something we can fix, so that even if the lag persists at least it will not interfere so severely with breeding dragons. Thanks and sorry for the frustration this has been causing!

I can confirm that based on our logs, there have been intermittent issues regarding Auction House listings taking a while to show up in search results. We are in the process of investigating the cause.

Hello all,

Just wanted to update that we are looking into this issue.

At the moment, one current lead we are following is that we are seeing indications that we have increased packet loss in the evenings (around 17:00-19:00 Server Time, give or take) based on charts we record. Based on similarity to past issues, this indicates it may be in networking equipment outside our servers, and it also seems like it may be dependent on time of day.

We have some staff members that are also experiencing this issue when it occurs and hopefully that will help us gather information we can send to our hosting provider.

We are also seeing some issues that relate primarily to our search backend, they could be connected or they could be unrelated, but the overlapping timing is suspicious. One example is that there have been instances where new Auction House listings were taking a while to show up.

I apologize for the frustration and inconvenience this is causing!

Hello all,
You cannot uncover any tiles while there is work order (Rugged Work or Precise Work) in progress on that specific dig site. Speaking with the team, it looks like this was discussed but would have complicated the implementation of the feature as there are already a lot of moving parts.

This has been fixed, thank you!

@mithrel We have a tentative fix for this issue now, can you let me know if it works?

We did not change the character limit, but we did run into an issue that happens to be triggered by character counts over a certain number.

@Illusia We have made a potential fix, can you check if it is working for you now?

Hello @Natron @Foa Can you check if it is fixed now? Thanks!

Hello @Illusia, we are working on a fix for this issue right now! It should be fixed fairly soon.

@Natron
Can you confirm this occurs when you actually go to submit your changes after editing a post, to make them live?

UPDATE: I think I have found the issue, I am working to resolve it.

@Yewwily @Disillusionist Thank you for the reports, this has been fixed now!

A brief explanation of what happened is in this post: https://www1.flightrising.com/forums/bug/3090079/7#post_49926233

Sorry about that everyone! We fed all of the dragons on the site several days ago and that resulted in everyone receiving a Well-Fed Bonus last night. Certain intensive processes that are supposed to run only during rollover ended up taking longer because of this, and they ran into the time period when players have returned to the site. As a result we had a few hiccups, including this issue.

Everyone has been reset to 10 Tomo turns again as of a few minutes ago.

Thanks for reporting this! We have made several changes to the Coliseum item drops for Frigidfin Expedition and extended the event, as detailed in the announcement thread.

Thanks for reporting this! We have made several changes to the Coliseum item drops for Frigidfin Expedition and extended the event, as detailed in the announcement thread.

Hello all,

After looking into this and speaking to our advertising partner, we believe the alerts about this URL are false positives. That is, the URL is not truly malicious but AVG and Avast are now reporting it as such. It is however an advertising-related script.

The reasoning behind that statement is that we know the script is part of our advertising partner's standard advertising stack and has been since at least early March 2020. We have records of this because it turned out to be unintentionally interfering with the functioning of certain buttons on the site around that time period, and we had to work through it with them. We are double checking with them what we can share about the specific function of this script.

In the mean time, however, we recognize that these pop-ups are both worrisome and confusing. We are communicating with our ads partner about the issue.

Clearing your cache may help in some specific cases as well and is always worth trying if nothing else is working. I have updated my recent post to mention this.

Hello all,

We have made a configuration change to our servers that may help the 400/Bad Request error problem for now.

Based on our monitoring information, it looks like the Cookie header in some players' HTTP requests is going over an internal limit. In other words, some cookie is getting too large. This started around the morning of the 24th, and we did not make any software deployments at that time. It is likely that this is coming from an advertisement or our ad partners' ad serving scripts, but it is difficult to track down as we would need retroactive logs of the typical sizes of all incoming cookies from before the event started.

We have doubled the limit for incoming cookie headers on our Web servers. Unfortunately, this is not an ideal solution, as the limit we had is pretty standard and you may now instead run into limits on browsers, intermediate proxies or firewalls, etc. Additionally, some of our Fairgrounds games and Coliseum may still be affected by this issue. Another problem is that if the cookie that is growing continues growing unbounded, this may just occur again soon, at which point we cannot reasonably raise the limit further.

If you continue getting this issue, for now the recommended course of action is to clear your cookies for flightrising.com and/or www1.flightrising.com in order to start with a "clean slate." If this does not help, clearing your cache is also recommended.

Thank you for your patience while we continue to investigate the root cause of this issue!

Hello everyone! Thank you so much for your patience and understanding yesterday. This maintenance did not go as planned, and we apologize for the inconvenience and any frustration it may have caused you. Due to how long this maintenance lasted and as many of you noticed last night, your dragons were fed before we opened the site back up to players.

So what happened?

In our efforts to improve the performance of Flight Rising during heavy load, such as during breed releases or anniversaries, we contacted our hosting provider about upgrading our internet link speed. In order to do this, our front-end server was going to be moved to a rack that supported the higher speeds. This was scheduled to take approximately 1 hour.

Unfortunately, a number of things went wrong during the move:

• Our hosting provider did not inform us that the move would cause our primary IP address to change, or that there would be no way to change it back besides returning the server to where it started. As a result, we had to update our DNS settings.
• At some point in the move or the subsequent debugging, some of the boot settings on the server became corrupted, and we had to restore them to their correct values so the server would boot.
• And the most serious issue: during the move, the server's secondary network port, which is integrated into the motherboard and is used to communicate with other back-end servers, was physically damaged. As a result, we could not open up the site because we could not communicate from this server to any of our other servers.

The last point took the longest for our hosting company to diagnose. At present we are now instead using a third network port which is part of an add-on card. We will be closely monitoring performance to see if there are any degradations relative to the previous setup.

What does this mean?

Some players may experience difficulty accessing the site, or parts of it, while the DNS changes propagate. These issues should resolve relatively soon as the changes were made almost a full day ago.

Thank you again, for your patience and your understanding during yesterday's surprise extended maintenance. It was an interesting day for everyone across the board.

Sincerely,
The Flight Rising team