v-CA Webpage slow to load and hang in system when entering PCB ID
CustomerNot indexed
CompanyNot indexed
Priority2
TypeAOI - No Action Required
UpdatedAug 1, 2026 at 6:24 AM
Thread156 messages
Ticket metadata
Group
AOI US (150000419797)
Status
Closed (5)
Created
Apr 1, 2026 at 3:50 PM
Requester
Not indexed
Company
Not indexed
Last synced
Aug 3, 2026 at 7:58 AM
Resolved
Jul 22, 2026 at 1:26 AM
Closed
Jul 30, 2026 at 5:26 AM
Tags
A - AOI CASE ACK EN, A - Reply Open, A-REOPEN, LOCKED
Current case
Customer issue
Omron Team, We are currently seeing a reoccurring issue here today on our S730 AOI server. 1. The webpages for services on the server (v-CA, v-DB, Q-up Navi) are loading slowly or not loading at all. 2. For v-CA (this is the only one I have looked at personally) if the webpage does load, then when the operators are entering the PCB ID, there is a hang waiting for the data to load. A frozen loading wheel appears on the screen and it is taking minutes for the data to load. We have not received any reports of issues with the S1080 AOI server. Has the Omron team seen this before? What logs are necessary to pinpoint the cause of the issue? Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
Conversation
Engineer and customer thread
156 messages
C
CustomerCustomer reply | Apr 1, 2026 at 4:17 PM
We have also had twice today where the operators were not able to login at all. I restarted the server for the first time and on the second time I restarted to Apache Tomcat service. They were able to login then but they could not load anything in v-CA so I ended up restarting the server again. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Hello Nolan, While it sound like data is being processed, data maybe only trickling through. Are there any active firewalls or AV software that may be scanning data as t's being transferred? Are all stations having this issue (vCA/vTS/Qup)? Please check the SETTIME batch files to confirm prism user and passwords are correct. Thanks,
freshdesk-inline-150156024466.png
C
CustomerCustomer reply | Apr 1, 2026 at 5:22 PM
Tony, Yes, this is affecting all stations when it does happen. Matt will have to check the other items in your email. I do not have access to that. Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
4 tiny or repeated images hidden.
C
CustomerCustomer reply | Apr 1, 2026 at 5:53 PM
No firewalls and AV software is installed but no indication it is scanning anything. Just received word that it is happening again. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Adding Sam Bowers to the email chain. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Hello Nolan, Can you please check the time zones on those PC's. Makes sure all time zones are the same Thanks,
freshdesk-inline-150156029657.png
C
CustomerCustomer reply | Apr 1, 2026 at 6:19 PM
Good Evening, This event is currently being reported as production down to the business. We need to get on a call to troubleshoot this further, especially before we start getting into third shift. Can someone be available for a call this evening? Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
14 tiny or repeated images hidden.
C
CustomerCustomer reply | Apr 1, 2026 at 6:34 PM
EOC is starting a bridge for us to join. An invite will be getting sent out. Tony I know you said you were on another call. We can invite you when you are available. If there is another engineer that would be able to join sooner we can add them as well. Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Hello Nolan, I am working to see if someone else can assist with this call. On the AOI server can you please check that Apache tomcat services are running. I know previously you stated that restating that service helped, can you please try restarting. Also has there been any recent updates that be affecting this service. Thanks,
freshdesk-inline-150156032374.png
C
CustomerCustomer reply | Apr 1, 2026 at 6:57 PM
Every time the issue has occurred the Apache Tomcat service has still been running. I have never seen it as stopped. This afternoon I restarted the service and we could login to the app but they still could not load anything. At this time the server has been restarted and it is working but this is the third time today. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Hey Tony, Have you heard back if Franciso is available? Let us know and we can send the invite to the bridge. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Hello Nolan, Can you or your team please do the following; 1) log into vDb tool, when you log in do you see all services as OK, are any NG. 2) Do you see any NG under Error occurrences are any NG? Please note any that are NG 3) Go to Maintenance - Log Collection and collect logs for vCA and System monitor Thanks,
freshdesk-inline-150156035568.png
C
CustomerCustomer reply | Apr 1, 2026 at 8:43 PM
And attached are systemmonitor logs Two emails for file size limits
freshdesk-inline-150156038505.png
26 tiny or repeated images hidden.
C
CustomerCustomer reply | Apr 1, 2026 at 8:44 PM
Tony, Sorry for the delay. Right as you requested those files we had another event where everything went down. Once again, the web page for vCA would not load or it would load slow. If it did load, then we can sign into the server without any issues. Once we go to enter a PCB ID to perform an inspection, the inspection results would not open; the webpage remains frozen on the enter PCB ID page. The status of the services show as OK. [email] Error Occurrences show as okay also. [email] Also I was able to get into vDB after waiting for a few minutes for the webpage to load during the event. Even during that the statuses and error occurrences showed as okay. See the attached vca logs. Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States Attachment(s) that exceed the 20 MB limit have been dropped. Please reach out to the sender.
freshdesk-inline-150156038556.png
26 tiny or repeated images hidden.
C
CustomerCustomer reply | Apr 1, 2026 at 8:57 PM
Tony, Sorry for the delay. Right as you requested those files we had another event where everything went down. Once again, the web page for vCA would not load or it would load slow. If it did load, then we can sign into the server without any issues. Once we go to enter a PCB ID to perform an inspection, the inspection results would not open; the webpage remains frozen on the enter PCB ID page. The status of the services show as OK. [email] Error Occurrences show as okay also. [email] Also I was able to get into vDB after waiting for a few minutes for the webpage to load during the event. Even during that the statuses and error occurrences showed as okay. Third attempt at sending this follow up. Ran into issues with sharing the logs. Sending the systemmonitor and vca logs through secure file transfer. I included you and Jeremiah because Jeremiah is joining us for the support call. Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
Hello Nolan, I just got back in. I am reviewing the logs now. Is there a change for remote access? I can't remember if your site allows it or not?
OA
Omron Application EngineerOmron reply | Apr 2, 2026 at 1:01 AM
Hi Team, I wanted to provide a quick update following our troubleshooting session. We performed a detailed review of the system, including application logs, Oracle logs, and server event logs, to better understand the behavior observed during the vCA hang condition. From our investigation, we were able to confirm that the Oracle database experienced a restart/recovery event during the timeframe of the issue. This aligns with the connection errors seen in the application logs at that time. After the restart, the database and related services returned to a normal operating state. At this point, the system appears stable, and we did not observe any active session or process limits being reached on the Oracle side during our checks. However, there are indications that system-level conditions (such as resource pressure) may have contributed to the event. To ensure we fully understand the root cause and determine if any preventive actions are recommended, we have submitted all collected logs and findings to Omron Japan for further analysis. We will follow up as soon as we receive their feedback. In the meantime, please continue to monitor the system and let us know if the issue reoccurs. Thank you for your time and cooperation during the investigation. Best regards,
OI
Omron internalPrivate note | Apr 2, 2026 at 1:02 AM
Private
https://gects.isd.iab.jp.omron.net/AGEWeb/ProgressManagementInfo/Details.aspx?InquiryID=260009 Ageweb inquiry above. Please help follow this and report back to the customer. I am at Denso tomorrow. Thanks.
C
CustomerCustomer reply | Apr 2, 2026 at 8:25 AM
Hi Jeremiah, Appreciate the update and the time you are taking to assist us with this issue! Since around 7:00 PM last night we have not heard of any issues. We will standby until we hear back in regard to what Omron Japan’s findings are with the Oracle database and recommendations. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
29 tiny or repeated images hidden.
C
CustomerCustomer reply | Apr 6, 2026 at 9:14 AM
Jeremiah, We have not had any issues since Wednesday evening but we still want to identify what happened and what we can do to prevent it. Have you heard anything from Japan yet? What are the next recommended steps? Thank you. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Omron Application EngineerOmron reply | Apr 8, 2026 at 12:59 AM
Hello Matt and BAE-IN team, I wanted to provide an update following our investigation and feedback from Omron Japan. Based on the logs and analysis, the root cause of the issue has been identified. The Oracle database was unexpectedly restarting due to repeated failures when attempting to write audit logs to the Windows Event Log. This caused the database to become temporarily unavailable, which resulted in the vCA hang behavior and the connection errors we observed. Omron Japan has provided a solution to resolve this by changing the audit log destination back to within Oracle. This prevents the write failures and should eliminate the restart behavior. In addition, they also recommended updating Oracle’s memory configuration to better align with the current system (64GB RAM), which will improve overall performance. This is not directly related to the restart issue, but is a recommended optimization. At this time, we recommend performing these changes during a planned production downtime window with Omron support present. While the steps are relatively straightforward (running provided scripts and restarting the Oracle service), they modify core database configuration and should be executed in a controlled manner to avoid any unintended impact. We will coordinate with you to schedule a suitable time and ensure the changes are applied safely. Please let us know your availability, and we can proceed with planning the implementation. Best regards,
OI
Omron internalPrivate note | Apr 8, 2026 at 1:05 AM
Private
Adding files from Omron JPN for root cause fix for BAE-IN. If somebody is able to help them before I can get back to them, here are files + howto.
C
CustomerCustomer reply | Apr 8, 2026 at 12:02 PM
Jeremiah and Omron Team, Thank you for the reply. We encountered the issue again today at 11:10am. This is the first report of reoccurrence since the last server restart at ~8pm EST last Wednesday (4/1/2026). We need to schedule this as soon a possible. We are available with normal hours until 5pm EST typically but we can support after that as well if that is when the Omron team is free. How long is the downtime for making the changes estimated to take? What is the Omron Team’s availability to support it? Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
28 tiny or repeated images hidden.
C
CustomerCustomer reply | Apr 8, 2026 at 3:35 PM
Brad, Can you help in scheduling FSE support to make the needed server / oracle changes recommended by HQ? We would like to get these changes implemented at your earliest availability. We are available 8:00 am until 5:00 PM EST. We would like to complete this tomorrow (Thursday) if possible. Thanks Tom Warnecke Principal Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions M: [phone]. | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States Connect with BAE Systems: [email] [email] [email] [email]
Hello Nolan, This will only take a couple of bat files to be ran, to make simple changes to Oracle. This will literally be less than 2 minutes, BUT your production needs to be stopped completely for AOI/vCA/vTS. Just FYI
C
CustomerCustomer reply | Apr 8, 2026 at 4:23 PM
Jeremiah, That sounds great. Would you be available tomorrow at 11:30 AM EST. If so, we can shut all systems down so that you can make the changes. Just to clarify, “production needs to be stopped completely for AOI/vCA/vTS” Does that mean the systems can remain on just not actively run or does that mean all application needs to be closed? I assume everything can remain powered on? Thanks Tom Warnecke Principal Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions M: [phone]. | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States Connect with BAE Systems: [email] [email] [email] [email]
Hello Thomas, All machines can stay logged on and powered up, but just needs production to not run any product, or review anything. If BAE-IN is comfortable doing this, then please go ahead. I just wanted to make it clear that production needs stopped. If it is not stopped, then data could be lost. Let me know what you decide, and if you rather wait for me, that's fine. Also, I am on a flight back home tomorrow at 10am Central, so unfortunately I will not be available. Attached are the files with the instructions. Thanks,
C
CustomerCustomer reply | Apr 8, 2026 at 4:55 PM
The zip files were removed from the email. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Hello Nolan and Thomas, I have uploaded the files via BAE-IN Secure File Transfer. Thank you.
OA
Omron Application EngineerOmron reply | Apr 10, 2026 at 9:15 AM
Hello Nolan and team, Can BAE-IN confirm if they implemented that Oracle fix? Or do we need to connect to get this done? Let me know?
C
CustomerCustomer reply | Apr 10, 2026 at 9:17 AM
Jeremiah, We implemented the oracle fix yesterday. Thanks for checking in. Tom Warnecke Principal Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions M: [phone]. | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States Connect with BAE Systems: [email] [email] [email] [email]
18 tiny or repeated images hidden.
C
CustomerCustomer reply | Apr 10, 2026 at 9:19 AM
We implemented the fix. The main difference we saw was the server memory usage has increase from 18 GB to now 39 GB. Other than that everything seems fine. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Omron Application EngineerOmron reply | Apr 10, 2026 at 10:15 AM
Hello Team, Great news. Please let us know if you have any other issues with this! I will resolve and close this ticket.
C
CustomerCustomer reply | Apr 20, 2026 at 1:03 AM
Hi Everyone, It has been reported Omron went down again roughly around 11:58 PM on 4/19 with similar symptoms as before, webpage was inaccessible. Rebooted the server (AQUSFWAOMAOI) and confirmed Omron is accessible again by Dave Salgado. We need to dig into this further. This is at least the 4th production down situation this has caused. **Production is currently running** [email] – What are our next steps? Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
20 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | Apr 20, 2026 at 9:50 AM
Hello Sam, Can you help collect new logs so I can relay this back to Japan? I'll need two sets of logs. First is below, from the Log Collection menu in vDB. Second log I'll need is in --> E:\oracle\diag\rdbms\prismdb\prismdb\trace\alert_prismdb.log < -- Can you please send both logs to me? Thanks,
freshdesk-inline-150158404681.png
C
CustomerCustomer reply | Apr 20, 2026 at 10:12 AM
Jeremiah, I sent the log files to you via Secure File Transfer. Thanks, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
17 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | Apr 20, 2026 at 10:14 AM
Hello Nolan, Thank you. I have received the files and will be reviewing them, and updating Japan inquiry we already have open.
OA
Omron Application EngineerOmron reply | Apr 20, 2026 at 10:35 AM
Hi Team, As part of our continued investigation into the recent Omron / Oracle outage, we have identified a specific timeframe where the database experienced an unexpected restart. From the Oracle alert logs, the failure sequence appears to begin around: ~00:31 AM on 4/20 During this time, Oracle reported: Audit write errors (OS-level) Followed by a crash/restart sequence A “VKTM detected a time stall” warning during recovery (which can indicate a system or VM pause/stall condition) Since the Oracle server is running in a VMware environment, we would like to check if any infrastructure-related activity occurred at or around that time. Could you please review VMware / host-level logs for the following timeframe: ~00:25 AM – 00:40 AM on 4/20 Specifically, we are looking for any of the following: VM snapshot operations Backup jobs (Veeam or other tools) vMotion / live migration activity Host resource contention (CPU, memory, ballooning, swapping) Storage latency or datastore issues Any VM pause, freeze, or time synchronization events Any information you can provide from the hypervisor or infrastructure side during that window would be extremely helpful in determining whether the Oracle interruption was influenced by system-level conditions. Please let us know what you find. Thank you,
C
CustomerCustomer reply | Apr 20, 2026 at 11:03 AM
That VM was being backed up by Veeam at that time. It started at about 12:15 AM and finished around 1:16 AM Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
But multiple other times with this issue nothing was going on with the host. And it happened again here just before 11 AM when I sent my previous reply. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Omron Application EngineerOmron reply | Apr 20, 2026 at 11:48 AM
Hello Matt, Can you please send me a round of new logs for this 11am crash? Second log I'll need is in --> E:\oracle\diag\rdbms\prismdb\prismdb\trace\alert_prismdb.log < -- I wanted to share a quick clarification based on our latest review of the Oracle logs. We can confirm that the audit logging change provided by Omron Japan has been successfully applied and remains in effect. Oracle is now writing audit logs internally (not to the Windows Event Log), so the original audit-related issue has been addressed. However, since the system experienced similar behavior again, both during the backup window overnight and again this morning around ~11 AM, this indicates there is an additional contributing factor beyond the audit logging configuration. From the Oracle logs, we are seeing signs consistent with brief system-level interruptions (including time stall behavior), which can impact database availability even when Oracle itself is configured correctly. The backup activity may contribute to this under load, but the recurrence outside that window suggests a broader condition. I'd like to investigate that. (Collecting more info for Japan to review tonight during their work hours) Thanks,
Hello Matt, Nolan got me the 11am logs. From the Oracle alert log during the ~11 AM event, we can see that the database did not crash unexpectedly. Instead, it performed a controlled shutdown (“shutdown immediate”) followed by a restart. This is an important distinction, as it indicates the database is being stopped externally rather than failing internally. Given this behavior, we would like to confirm if anything on the system or infrastructure side could be triggering a restart of the Oracle service. Could you please check for the following around the ~11 AM timeframe: Any Windows Service restarts (OracleServicePRISMDB) Scheduled tasks or scripts that may interact with Oracle Backup, monitoring, or security software actions Any system-level restart, patching, or automation VMware guest or host-level actions affecting the VM Even if nothing was intentionally triggered, we want to confirm whether any automated process or policy may be interacting with the Oracle service. This will help us determine what is initiating the shutdown sequence we are seeing in the logs. Please let us know what you find?
C
CustomerCustomer reply | Apr 20, 2026 at 12:09 PM
At about 11:05 we restarted the server to resolve the issue. It had been happening since just before 11. No other incidents or changes had happened. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
And happened again around Noon. Will send more logs Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Jeremiah, Sent another set of logs from the noon event. Thanks, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
Hello Team, Can somebody take this text file I emailed you, change it to a ".bat" file, then run it on the desktop of the server? It will generate a text file. Can you email that back to me? oracle_os_check.txt -> oracle_os_check.bat
C
CustomerCustomer reply | Apr 20, 2026 at 12:26 PM
Jeremiah, would it be easier if we setup a screen sharing session to run through these troubleshooting steps? Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
20 tiny or repeated images hidden.
C
CustomerCustomer reply | Apr 20, 2026 at 12:27 PM
Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Hi Matthew, Thank you again for your help with the troubleshooting so far. We completed the same extended OS-level diagnostic on a known-good AOI / vDB server in our environment and did not observe any EvntAgnt events or event log access issues there. That gives us a cleaner baseline for comparison against your server. At this point, we would like to collect the same extended diagnostic output from your system so we can continue narrowing down the remaining OS-level behavior we are seeing. Please run the attached file (again change from .TXT to .BAT) oracle_os_check_extended.bat This will generate a timestamped .txt output file in the same folder. Please send that output file back to us after it completes. This script collects: Windows Event Log service status Event log configuration Disk space and system information Time synchronization status Oracle-related services Recent System/Application warnings and errors EvntAgnt event checks Event log read tests Agent/service/program discovery related to logging/monitoring tools One item that still needs to be checked manually on your side is CrowdStrike / security policy review. In particular, we would still like confirmation on whether the following are excluded or whitelisted: C:\Windows\System32\winevt\Logs Oracle-related directories (for example E:\Oracle\...) Oracle services/processes related to PRISMDB Please also let us know if CrowdStrike or any other security/monitoring policy is actively inspecting Windows Event Logs or Oracle processes. Once we receive the output file, we will compare it against our known-good reference system and continue the investigation. Thank you,
C
CustomerCustomer reply | Apr 20, 2026 at 2:18 PM
Reported that they are having issues again, started shortly before 2 pm. Server restarted at 2:16 pm. Ran the bat file shortly before server restart and it is attached. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
And run again after the server restarted. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Jeremiah, Sent another round of log files. I was told at 2pm that issues started 15 to 20 minutes before that. I don’t know how accurate that recount is though. Thanks, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
Hi Matt, Thanks again for sending over the latest logs, especially grabbing them before the restart, that helped a lot. I spent some time going through everything, and I think we’re finally getting a clearer picture of what’s going on here. At a high level, Oracle itself looks OK. The audit logging setting is still correct (it’s not trying to write to the Windows Event Log anymore), and the database behavior we’re seeing doesn’t point to a configuration issue on the Oracle side. What does stand out is what’s happening at the OS level. We’re seeing repeated “EvntAgnt” warnings where the system is having trouble opening or reading event log files. What’s important is that these aren’t just showing up after a reboot...they’re happening during normal operation as well. Along with that, there are also “Access is denied” warnings tied to security policy processing. To sanity check this, I ran the same diagnostics on one of our in-house AOI/vDB systems that we know is healthy, and we don’t see any of this behavior there — no EvntAgnt events, no event log access issues. So at this point, the pattern we’re seeing is: Oracle is running, but relying on OS-level logging The OS is intermittently having trouble accessing its own event logs That ends up impacting Oracle (timeouts, instability, vCA hanging, etc.) Given what’s installed on the server (CrowdStrike and Nessus), the most likely cause here is some kind of interaction between the security/monitoring tools and Windows logging or protected system resources. I don’t want to jump to conclusions, but this is definitely the direction the data is pointing. Could you have your IT/security team take a look at a few things on their side? In CrowdStrike, confirm whether the Windows Event Log path C:\Windows\System32\winevt\Logs and the Oracle directories/processes are excluded or allowed Check if Nessus or any other monitoring tools are scanning or interacting with event logs or Oracle files Take a look at the “Access is denied” (SceCli 1202) messages and see if there are any policy restrictions or conflicts there At this point, it doesn’t look like something we can resolve from the AOI / Oracle side alone ...it’s more about what’s happening at the OS/security layer. Thanks,
C
CustomerCustomer reply | Apr 20, 2026 at 3:16 PM
The Access is Denied (SceCli 1202) for security policy processing is always for a failure to open the Windows Defender Antivirus Service. Windows Defender is turned off by other group policy. The error comes from a group policy setting attempting to access it. That should not be related. The EvntAgnt warnings I’m only seeing immediately after the server is restarted. It does not appear any other time until again immediately after the server is restarted. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Hello Matt, Thanks for the clarification, that all makes sense on the SceCli and EvntAgnt points. At this stage, I agree those items by themselves don’t fully explain what we’re seeing. What stands out more from the latest logs is the behavior leading up to the issue itself. Right before the system becomes unusable, we’re seeing a clear pattern where operations that normally complete quickly start taking longer and longer (60s → 120s → 180s), eventually leading to timeouts and invalid responses. This is happening across multiple services at the same time, not just a single component. We also confirmed from the Oracle side that the database is not crashing or throwing internal errors, it’s still running, but becoming unresponsive under load. The restart is essentially recovering from that degraded state. One thing I do want to call out from the logs is that CrowdStrike components are actively loaded inside the Omron application process during runtime. That means it’s not just installed on the system, but actually participating in execution. I’m not saying this is definitively the cause, but given the pattern we’re seeing (progressive slowdown, multi-service impact, timeouts without a crash), it does line up with what we typically see when something in the OS or security layer is introducing latency or contention. At this point, the Oracle layer looks stable, so the next step would really be to rule out anything in that execution path. Would it be possible to either: confirm whether exclusions are already in place for the Omron/Oracle processes and event log paths, or temporarily relax monitoring/exclusions for those components to see if the behavior changes? That would help us either confirm or eliminate that variable pretty quickly. Thanks,
C
CustomerCustomer reply | Apr 20, 2026 at 6:20 PM
Reply from our Crowdstrike SME as follows: CrowdStrike does not actively scan like traditional AV products. Are there any logs that point to CrowdStrike as either blocking or prohibiting functionality? Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
No incidents last night or so far today. No changes were made on our side. Still wondering what the cause might be and how to prevent this. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Jeremiah, What should our next steps be. Our crowd strike team says that it does not actively scan like traditional AV products but also wants to know if there are there any logs that point to CrowdStrike as either blocking or prohibiting functionality? So far today there are no issues. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Hello Matt, Thanks for the follow-up, and I appreciate you checking with your CrowdStrike team. To answer your question directly, we are not seeing any logs that show CrowdStrike explicitly blocking or denying functionality. There is nothing indicating a hard stop or access being prevented. What we are seeing is that CrowdStrike components are loaded inside the Omron application process during runtime. At the same time, the system shows a pattern where operations gradually slow down over time, going from around 60 seconds to 120 seconds to 180 seconds, and eventually leading to timeouts and unstable responses across multiple services. So from our side, this does not look like something being blocked. It looks more like the system is slowing down or being affected during execution. That is why we still want to keep CrowdStrike, or any similar tool, in consideration since it is part of the execution path. Also, it is good to hear there have been no issues so far today. That will help us compare normal operation versus when the issue happens again. For next steps, here is what I would suggest: If the issue happens again, please try to capture the same logs before restarting the server if possible, and note the exact time when the slowdown starts. For CrowdStrike, we are not asking to remove it, but if your team can confirm that exclusions are in place for Oracle and Omron processes, or if there is an option to temporarily relax monitoring for a short test window, that would help us confirm whether it is contributing or not. Continue monitoring for now since the issue is intermittent. Comparing a good state versus a bad state will be important. I also wanted to let you know that I reached out to our Japan team last night with all of the updated logs and findings. I will follow up with them again tonight to push for review on their side as well. I would like to make sure we get another set of eyes on this in case they recognize anything from the database or platform side. We are getting close to narrowing this down. At this point it is about understanding what changes right before the slowdown begins. Please let me know if anything changes on your side, and I will keep you updated as soon as I hear back from Japan. Thanks!
OI
Omron internalPrivate note | Apr 21, 2026 at 11:21 PM
Private
https://gects.isd.iab.jp.omron.net/AGEWeb/ProgressManagementInfo/Details.aspx?InquiryID=260009 I'm on vacation, will somebody please follow up with this customer with what Japan is asking for?
C
CustomerCustomer reply | Apr 22, 2026 at 12:56 PM
From our Crowdstrike admin: Policies have been disabled on AQUSFWAOMAOI. A restart is required to unload drivers before testing. If the same operations gradually slow down over time continues to occur after the above setting is in-place the next step would be to temporarily remove CrowdStrike, restart the re-test. CrowdStrike components are loaded inside the Omron application process during runtime. CrowdStrike works with processes, so this expected behavior. As a last resort, I will provide information to generate diagnostic data using CrowdStrike’s tool. In this scenario it would be beneficial to have Omron’s data output as evidence. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Good evening, Server went down again, reported at 11:16 PM. Rebooted the server and production is running again. Jeremiah, Have you received anything from the Japan team on this? Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Omron Team, At what point do we rebuild the database/server to get this stable again? I know we are waiting to hear back from Japan, but we need to escalate this. Can only tell production leadership “we are looking into it” so many times before they lose trust. I am going to send out a Teams meeting to discuss and plan next steps. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Omron Application EngineerOmron reply | Apr 24, 2026 at 9:06 AM
Hello Sam, Is there a possibility to completely remove crowd strike. I'm not sure if rebuilding the database is going to help with issue if there is a security software like CrowdStrike is processing each file. Please see the response from Japan and they have for the windows log. Could you please upload the windows log files? Originally, memory exhaustion (heavy swapping) was occurring frequently, but there were no internal problems with Oracle itself between April 19th and 20th. CrowdStrike appears to be a mechanism that loads and processes within each application's process, inspecting all process communications and file I/O in real time. This increases memory and CPU usage. Around 0:31 on April 20th, it was confirmed that Windows internal communications (RPC) failed repeatedly, and the entire OS was in an abnormal state. The RPC communication failures occurred when Oracle's audit log was being written, and it is thought that the communication abnormality occurred due to the high load on the entire OS. Oracle was manually restarted at 0:34. Please obtain and verify the following: • Windows logs (system, application) • Please confirm that Oracle-related entries are excluded from CrowdStrike's exclusion settings. **The influence of security software is a strong possibility. Best Regards
C
CustomerCustomer reply | Apr 24, 2026 at 9:16 AM
Hi Saqib, Appreciate the quick response. We can collect and provide these logs. Reached out to our CrowdStrike team this morning to see if we can fully uninstall rather than just having it disabled. Currently waiting to hear back. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
21 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | Apr 24, 2026 at 10:03 AM
Hello Sam, Please upload the Windows log file when you get a chance. Please send the email to my individual email instead of AOI support after uploading the file. The AOI support email is common, and it is difficult to maintain passwords and logon information for the group. I appreciate your help. Best Regards
C
CustomerCustomer reply | Apr 24, 2026 at 10:15 AM
Sam briefly stepped out but I just sent the logs to your email. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Crowdstrike has been uninstalled from that server. Matthew Lewis Systems Admin Sr VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Omron Team, We had an event today at approximately 1:45pm. Log files have been sent via secure file transfer to Brad, Jeremiah, Jeremy, and Saqib. Once I was made aware of it, I check v-DB to see how many people were connected. V-DB took a bit longer to load than usual but it did load. * 13 for v-CA * 2 for qup Navi Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
Good afternoon, Wanted to follow up in regard to see if anything was found from the logs from our latest event. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Sam, We have validated that there is a spec on limit of web app connections on a single server. It seems to be 10 simultaneous connections. This relates to v-CA, Q-upNavi and v-DB. We are wondering if this may be the actual root cause at this point. We may be able to set up a line / master server configuration to partition the traffic more effectively, but we will be interfacing with the development team for next steps. What is the maximum number of simultaneous connections that BAE may require at any given time? @Jeremiah Soliz / OEI - any additional thoughts? Best regards, Jeremy Orbach Application Engineer II – Inspection Systems Direct: [phone] | [email] Omron Automation Americas 2895 Greenspoint Parkway, Suite 100, Hoffman Estates, IL 60169 automation.omron.com ________________________________
Hi Jeremy, We just had another incident of the server going down just now. Rebooting the server allowed production to resume. I would refer to either Tom or Nolan, they can give a more accurate count than what I can. Is that normal for an Omron setup of 10 or more connections to bring down the server like that? Can we escalate the communication with the development team? This server has went down close to 10+ times, wish I was exaggerating. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Sam, As far as I am aware, BAE has more simultaneous v-CA connections than any other customer who does not already have the line / master setup. Please monitor if performance degrades after 10 simultaneous users, as I am reading that the CPU may cap at 100% when too many requests and coming in for Tomcat. Larger customers typically use our physical hardware in this configuration to ensure performance. If 20+ connections are necessary, likely 3 line servers + 1 master server would suffice. Clearly the 10 limit is not a hard limit, but if issues are ALWAYS isolated to 10+ users, I believe this is the next best step to ensure we fully eliminate this issue from persisting. Can logs be provided for the event from tonight? Best regards, Jeremy Orbach Application Engineer II – Inspection Systems Direct: [phone] | [email] Omron Automation Americas 2895 Greenspoint Parkway, Suite 100, Hoffman Estates, IL 60169 automation.omron.com ________________________________
Hi Jeremy, Appreciate the insight. We can get with the team tomorrow and make these simultaneous connections and monitor performance to see if it follows the working theory. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Server just went down again. Is there a way to limit connections? Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Sam, I am not aware of one. Did the CPU max out? How many connections are trying to connect currently? Is there anything different about today vs. last week? I thought we were stable last week after disabling CrowdStrike. Please provide logs for all related events from today. Best regards, Jeremy Orbach Application Engineer II – Inspection Systems Direct: [phone] | [email] Omron Automation Americas 2895 Greenspoint Parkway, Suite 100, Hoffman Estates, IL 60169 automation.omron.com ________________________________
Hi Jeremy, CPU usage is not spiking to 100% during these crashes. I think everyone else is offline, how can I view the number of active sessions? Unfortunately, does not seem to be any change since uninstalling CrowdStrike. We can get the logs over to you, it may not be until tomorrow before we can. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Sam, Login to v-DB. Under user settings scroll down to the bottom and you will see active connections. Best regards, Jeremy Orbach Application Engineer II – Inspection Systems Direct: [phone] | [email] Omron Automation Americas 2895 Greenspoint Parkway, Suite 100, Hoffman Estates, IL 60169 automation.omron.com ________________________________
Jeremy, Currently there are 16 connections to v-CA. We were not stable last week after uninstalling CrowdStrike. As was sent on 4/29/2026, we had to restart the server again. There were 13 connections to v-CA and 2 connections to Q-upNavi. As far I am aware there is not anything different today than last week. CrowdStrike is still uninstalled and the number of connections to the server is about the same. I will work on getting the logs from the server with Sam to send to the Omron Team. Thanks, Nolan
Jeremy, Copying info previously sent to keep the email chain easy to read. Currently there are 16 connections to v-CA. We were not stable last week after uninstalling CrowdStrike. As was sent on 4/29/2026, we had to restart the server again. There were 13 connections to v-CA and 2 connections to Q-upNavi. As far I am aware there is not anything different today than last week. CrowdStrike is still uninstalled and the number of connections to the server is about the same. I also sent the Log files from today via secure file transfer for the Omron team to review. Thanks, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
This is the CPU usage from 5 pm to 9 pm last night which covers the time of both incidents [email] Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
On the flip side, is it unusual for the CPU utilization to be that low? Where the database needs to be optimized to fully take advantage of the server resources? Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Jeremy and the Omron Team, I am certain we (BAE) discussed the number of simultaneous v-CA connections before we purchased the Omron AOI platform. The exact # was not known but we were told that 30 would not be an issue. Also, we have been using this platform since 2019 and using software version 3.04 since June of 2024 without any issues until March of this year. With more simultaneous v-CA connections than we currently have today. I don’t believe this is the issue but if it is Omron we need to find a solution in a timely manner. There have been no updates to the Omron’s software since the upgrade to 3.04 in June of 2024. However, the server is updated every month. Analytically speaking, it is logical that something with one of the updates has caused this issue. The only other thing that may have changed is the version of Fire Fox. Regardless of what the issue is, we are counting on Omron to determine the root cause of this issue. Have you received any word from HQ from the lasted log files sent? How can we expedite the resolution of this issue? Would there be any benefit from bringing an FSE onsite? Thanks Tom Warnecke Principal Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions M: [phone]. | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States Connect with BAE Systems: [email] [email] [email] [email]
We had the same issue again today at approximately 2:13 and service restored with a server reboot Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Logging on now to reboot. Sorry was driving home. Sam Bowers SPV II - Factory Support IT Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [Image] [Image] [Image] [Image] [Image] [Image] [signatureImage][signatureImage][signatureImage][signatureImage][signatureImage][signatureImage] ________________________________
Folks, I will be unable to attend tomorrow. My recommendation is to look towards different hardware and/or reconfigure the virtual server into master / line. To confirm, how many processor cores have been allocated to the server in question? Best regards, Jeremy Orbach Application Engineer II – Inspection Systems Direct: [phone] | [email] Omron Automation Americas 2895 Greenspoint Parkway, Suite 100, Hoffman Estates, IL 60169 automation.omron.com ________________________________
I could be mistaken, I don’t think anyone from Omron has looked at the server while the issue is occurring. Is there someone we can call directly the next time this happens to investigate further? Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Sam, Understand that this week is Golden Week in Japan, which means our technical support is currently limited, but will be back up to full speed shortly. We have had many calls and meetings on our end and can say decisively that something in the network infrastructure is causing this issue. If CPU, RAM utilization and network bandwidth at the time of crashing are isolated out as root causes, changing infrastructure may be necessary. Screenshots from the server of Task Manager showing load of all relevant modules when not functioning normally would isolate the physical hardware from the equation. From what I have seen, it appears as though the hardware never hits 100%. I do believe, based on the history of documented issues relating to v-TS losing network connection that moving to a physical server and/or line/master configuration would ultimately resolve these issues. Isolating our hardware from all updates and putting it on a segregated DMZ VLAN has kept it functional at companies like BAE and passes all ITAR compliance regulations. As Tom noted, updates are an uncontrolled factor that may or may not relate to the root cause but should be disabled for this exact reason. Relating to your second question, our support staff changes regularly, and this specific issue requires specialized support. We are happy to support via e-mail and can come on-site to troubleshoot with test hardware if further services are necessary. A quote would need to be created if this support is desired. We also normally provide this type of support during normal business hours (8 - 5PM CST). We do, however, understand the urgency of the issue. Given how much time this issue has consumed, I would advise getting physical master / line server configurations setup isolated from updates and the network in general if possible. Once Golden Week ends, we can confirm this with the development team. Thanks for your understanding. Best regards, Jeremy Orbach Application Engineer II – Inspection Systems Direct: [phone] | [email] Omron Automation Americas 2895 Greenspoint Parkway, Suite 100, Hoffman Estates, IL 60169 automation.omron.com ________________________________
The plant had another production down incident this evening. Reported to have started around 11:30 pm eastern. The server was restarted at 12:10 Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Omron Application EngineerOmron reply | May 6, 2026 at 1:02 AM
Hello Team, Can somebody zip up the Tomcat files for us to review? C:\Program Files\Apache Software Foundation\Tomcat 9.0\logs < -- this location (or Tomcat 6.0) on the AOI server. Thanks,
C
CustomerCustomer reply | May 6, 2026 at 10:58 AM
Working on gathering these logs. Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
44 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | May 6, 2026 at 12:02 PM
Hello team, Can you guys grab these files as well? Just grab those directories and zip them up please.
freshdesk-inline-150160648516.png
C
CustomerCustomer reply | May 6, 2026 at 12:08 PM
I’ve sent them to Nolan to upload. Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Omron Application EngineerOmron reply | May 6, 2026 at 1:08 PM
Hello team, Can somebody load vCA up, and check this version for me? I want to confirm what version vCA is at?
freshdesk-inline-150160661498.png
C
CustomerCustomer reply | May 6, 2026 at 1:11 PM
See image below [email]
freshdesk-inline-150160662214.png
37 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | May 6, 2026 at 1:21 PM
Hello Nolan, Can you run this command and paste the output back to me here? Do this from a command prompt on the AOI server ---> cd /d "C:\Program Files\Apache Software Foundation" findstr /S /I /N "3.03.00C 3.04.00C 7.03.00C 7.04.00C" *.xml *.txt *.config
C
CustomerCustomer reply | May 6, 2026 at 1:27 PM
Tomcat 6.0\config\admintoolConfig.xml:25: Ver.3.03.00C Tomcat 6.0\config\Version.txt:1:3.03.00CTomcat 6.0\config\vtwebapi.xml:23: Tomcat 6.0\config\vtwebapi.xml:24: Tomcat 6.0\config_bak\admintoolConfig.xml:25: Ver.3.04.00C Tomcat 6.0\config_bak\Version.txt:1:3.04.00CTomcat 6.0\config_bak\vtwebapi.xml:23: Tomcat 6.0\config_bak\vtwebapi.xml:24: Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Omron Application EngineerOmron reply | May 6, 2026 at 1:31 PM
Hello Nolan, Please run: --------------- cd /d "C:\Program Files\Apache Software Foundation\Tomcat 6.0" dir config dir config_bak dir /T:W config dir /T:W config_bak fc config\Version.txt config_bak\Version.txt fc config\vtwebapi.xml config_bak\vtwebapi.xml fc config\admintoolConfig.xml config_bak\admintoolConfig.xml ------------ Then paste that back to me? Also, please run: powershell -NoProfile -Command "Compress-Archive -Path '.\config','.\config_bak' -DestinationPath '.\BAE_Tomcat_config_compare.zip' -Force" And send that to me?
OA
Omron Application EngineerOmron reply | May 6, 2026 at 1:37 PM
Hello Nolan and team, Please join --> https://teams.microsoft.com/meet/44052390853225?p=diZ0xJFRA5jB5anHJX < -- I am here now.
C
CustomerCustomer reply | May 6, 2026 at 1:42 PM
Nolan does not have admin access to the server but I do. the powershell command returned an error PS C:\Users\sa_matthew_lewis5> powershell -NoProfile -Command "Compress-Archive -Path '.\config','.\config_bak' -Destina tionPath '.\BAE_Tomcat_config_compare.zip' -Force" Compress-Archive : The path '.\config' either does not exist or is not a valid file system path. At line:1 char:1 + Compress-Archive -Path '.\config','.\config_bak' -DestinationPath '.\ ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (.\config:String) [Compress-Archive], InvalidOperationException + FullyQualifiedErrorId : ArchiveCmdletPathNotFound,Compress-Archive Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Jeremiah, So far no reported issues with any slowdowns on v-CA. Here’s a new tomcat page as requested after we have been using the system for a bit. Thanks, Nolan
Good morning, Tomcat status as of this morning is attached. Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Omron Application EngineerOmron reply | May 7, 2026 at 8:44 AM
Hello Sam, Thanks for the PDF, but could you send the Complete Status one? http://localhost:8080/manager/status/all was the link I used from the AOI server.
C
CustomerCustomer reply | May 7, 2026 at 8:48 AM
Yes of course, please see attached. Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
20 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | May 7, 2026 at 7:52 PM
Hello Team, Can you please download the zip file I sent you? TomcatThreadDumpTool.zip Please it on the desktop of the AOI server. Unzip this file, then run Run_Tomcat_Thread_Dump.bat file in that zip file. This will copy the contents of this zip file to C:\Tools, then output a zip file to C:\temp once it's done collecting information. Once you do this can you please send me this zip file? Thanks!
C
CustomerCustomer reply | May 8, 2026 at 8:52 AM
I ran the file as requested. Nolan should be sending the zip file back to you. Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Jeremiah, I sent the resulting file to you via file transfer. Thanks, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
Omron Application EngineerOmron reply | May 11, 2026 at 8:27 AM
Hi Matt, Sam, Nolan, and team, Japan/development reviewed the latest information we sent over, including the Tomcat status captures and the Java thread dump data. At a high level, they are now looking at two areas: First, they found a Windows-side issue in the earlier event logs where the Software Usage Metrics database may be corrupted. This is Windows-related, not Omron-specific, but Japan believes it should be corrected because WMI-related retries can continuously consume system resources such as CPU and disk. They provided a cleanup step for this. Second, Japan confirmed that the long-running /vtweb/vca/IDSearchByBoard_vca requests we found are not normal. This function uses long-polling, but under normal conditions, the DB response should return in a few seconds to tens of seconds. They also confirmed that if many of these long-polling requests remain occupied, Tomcat threads can build up and new requests may not process correctly. They are still reviewing whether a Tomcat/vCA setting or a different server configuration is needed. They also noted that the session logs show roughly 20 to 30 unique user sessions per hour, which may be high for the current single-server setup. For the next step, Japan is asking us to collect two Windows security policy files from the vDB server. Please open Command Prompt as Administrator and run the following commands: mkdir C:\Temp type C:\Windows\security\logs\winlogon.log > C:\temp\winlogon_export.txt gpresult /z > C:\temp\gpreport.txt After running those, please send us these two files: C:\temp\winlogon_export.txt C:\temp\gpreport.txt Japan also recommended correcting the Software Usage Metrics database issue during a safe maintenance window. Before running this, I would recommend backing up the folder first. mkdir C:\Temp\SUM_Backup net stop SoftwareUsageMetrics xcopy C:\Windows\System32\LogFiles\Sum C:\Temp\SUM_Backup /E /I /H /Y del /Q C:\Windows\System32\LogFiles\Sum\* net start SoftwareUsageMetrics Please run this only during a safe window. It is not expected to rebuild the Omron database or affect vCA data, but since it is a Windows system cleanup step, I would still prefer it be done carefully. We will send the requested files back to Japan so they can continue reviewing the Windows security policy failure side. In parallel, they are still reviewing the Tomcat/vCA long-polling behavior and whether the customer load may require a configuration change or possibly a line/master server setup. Thanks, Jeremiah
C
CustomerCustomer reply | May 11, 2026 at 9:05 AM
Attached is the files requested. Please let me know if you get them. Nolan is out today and usually sends them. As for the second part, I got an error of “The service name is invalid” when trying the command of “net stop SoftwareUsageMetrics” My research indicated that “Software Usage metrics” is part of the User Access Logging Service. I was able to run the requested commands after stopping that service instead. Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Omron Application EngineerOmron reply | May 11, 2026 at 9:10 AM
Hello matt, Got the zip file, thank you.
C
CustomerCustomer reply | May 11, 2026 at 9:29 AM
Also, FYI, the Tomcat status page from this morning is attached. Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Good morning, Attached is the TomCat status page from this morning: Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Omron Application EngineerOmron reply | May 13, 2026 at 9:43 AM
Hello Sam, Thank you! I was literally about to email you and ask for this!
C
CustomerCustomer reply | May 14, 2026 at 9:20 AM
Of course! Have you heard anything else in regard to what next steps we should be taking for this issue? Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
27 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | May 14, 2026 at 10:34 AM
Hi Matt, Sam, Nolan, and team, Japan/development has provided another update, so I wanted to summarize where things stand. At this point, there appear to be two separate items being reviewed. The first item is related to Windows Defender / Group Policy. Japan reviewed the Windows security logs we provided and believes the Defender-related policy failure is not directly related to the vCA/server slowness. Their finding is that the current GPO named NHNA.WindowsDefender.Disable is trying to disable Windows Defender, but Windows Defender tamper protection may be preventing that change from applying cleanly. Since CrowdStrike has been removed, Japan recommends that your IT team review this GPO. Specifically, they recommend changing the WinDefend service setting and the DisableAntiSpyware registry policy setting to Not Configured. The second item is the vCA / Tomcat behavior. Japan confirmed that the /vtweb/vca/IDSearchByBoard_vca request remaining active is part of the long-polling design used by the “Board ID Reading Central Monitoring” screen. However, they also confirmed that Tomcat 6 can sometimes miss disconnect notifications, which can leave old threads active. If enough of these accumulate, Tomcat can eventually stop accepting new requests once the thread pool limit is reached. Japan noted that the data they reviewed still showed available thread capacity at that time, but this is the area we are continuing to monitor. Japan also stated they have not found past cases yet where the active Tomcat configuration remained at 3.03C after an upgrade to 3.04C, and they are not recommending Tomcat timeout/keep-alive setting changes because those changes may interfere with the long-polling behavior. For now, I have not asked your team to make the GPO changes yet because I wanted to review Japan’s response first and make sure we communicate the request clearly. The recommended customer-side action would be for your IT team to review the following GPO: NHNA.WindowsDefender.Disable Japan recommends reviewing these two settings: System Services Target: WinDefend Current: Startup Type = Disabled Recommended: Change to Not Configured Registry Policy Target: Software\Policies\Microsoft\Windows Defender\DisableAntiSpyware Current: Value = 1 Recommended: Change to Not Configured Separately, I would like to continue monitoring the Tomcat status to see whether the active thread count continues to climb. The key number we are watching is the Tomcat http-8080 busy thread count. If this approaches the 200-thread limit, vCA/vDB/Q-upNavi may stop accepting new requests. We will continue monitoring the daily Tomcat “Complete Server Status” PDFs your team has been sending. Those snapshots are very helpful because they allow us to track the active Tomcat thread count, vCA session count, and the long-running IDSearchByBoard_vca connections over time. We will continue collecting that trend data and sending it to Japan/development as needed so they can review whether the long-polling buildup is stabilizing or continuing to increase. Thank you,
C
CustomerCustomer reply | May 14, 2026 at 5:36 PM
Jeremiah, I personally have not had the opportunity to read through your response today. I would like the Omron team to be aware thought that we had another outage today this evening. I was told at 4:50pm that we had been down for an hour at that point. We started taking actions to correct at 5:12pm. v-CA logs, SystemMonitor logs, Event Logs, TomcatThreadDump, and Tomcat status page have been sent via file transfer for review. We restarted the server about 5:20pm. Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
18 tiny or repeated images hidden.
C
CustomerCustomer reply | May 14, 2026 at 7:20 PM
Server has gone down once again… Running the Tomcat dump tool to send additional logs. Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
24 tiny or repeated images hidden.
C
CustomerCustomer reply | May 14, 2026 at 7:32 PM
Attached is the Tomcat status page at the time of the crash. Nolan will be sending the Tomcat Dump logs when he is online next. Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
24 tiny or repeated images hidden.
C
CustomerCustomer reply | May 14, 2026 at 8:00 PM
Log files have been sent. Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
24 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | May 14, 2026 at 9:02 PM
Hello Nolan, Can you identify the device/server/user at IP address 10.62.99.21? During both May 14 outage captures, this IP was making many requests to: /prismRest/resources/GetPcbCompInfo These requests appear to be consuming a large number of Tomcat threads and may be the immediate cause of the outage. Please let us know? If you can stop this client, the Tomcat 200 thread pool will stop filling up. This is different then the previous Tomcat saturation outages. Let us know?
C
CustomerCustomer reply | May 14, 2026 at 9:09 PM
Jeremiah, Surface level look: it is not one of our review stations. I can look into more tomorrow once Sam, Matt, and I are back in plant tomorrow. Thank you Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
24 tiny or repeated images hidden.
C
CustomerCustomer reply | May 15, 2026 at 2:17 AM
10.62.99.21 is one of our bastion servers. Our personal computers are one network and the omron server and all production machines are on another. We use the bastion servers to connect from one to the other. We also remote into the Omron server from that server. Omron reported down again at 1:43 am I tried just restarting the tomcat service but the user reported that it did not help. Server restarted at about 2:10 Tomcat status, thread dump, and windows logs attached. I counted about 14 sessions in vdb before restarting Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Down again shortly after 4 am Restarted at 4:15 Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
And fresh event logs attached Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Good morning Jeremiah, Is this currently a high priority item for the Japan team? We understand the delay with the difference in time zones and there’s not much we can do until we hear back. I have to ask to make sure efforts are being maximized while we still have the patience of the plant. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [Image] [Image] [Image] [Image] [Image] [Image] [signatureImage][signatureImage][signatureImage][signatureImage][signatureImage][signatureImage] ________________________________
Omron Application EngineerOmron reply | May 15, 2026 at 8:44 AM
Hi Matt, Sam, Nolan, and team, I wanted to provide a more direct update on where we are with the vCA / vDB outage investigation and what we believe the next practical options are. Based on the latest Tomcat “Complete Server Status” captures and outage logs from May 14 and May 15, we are now seeing a much clearer pattern. The server is not simply “crashing” in the traditional sense. Instead, Tomcat is reaching, or getting very close to reaching, its available request-processing thread limit. When that happens, vCA, vDB, Q-upNavi, and prismRest can appear slow, hung, or unavailable, even though the Windows services may still show as running. This became much clearer once your team started sending the full Tomcat status PDFs before rebooting the server. Earlier in the investigation, we could see the symptoms in the application logs, Oracle logs, and Windows logs, but we did not have a live view of exactly which web requests were consuming Tomcat threads at the time of failure. The full Tomcat status page gives us that view, including the active request URL, client IP, request time, and Tomcat thread state. From the latest outage captures, we saw a large number of active requests to: /prismRest/resources/GetPcbCompInfo Many of these requests appear to be coming from the same client IP address: 10.62.99.21 So one clear next step is still for BAE IT to identify what system or application is tied to that IP address. It may be a reporting tool, script, dashboard, API process, or another automated process. If that client is making many GetPcbCompInfo calls at once, those requests can consume Tomcat threads and directly contribute to the outages. That item should be corrected by stopping, throttling, rescheduling, or modifying whatever process is generating those requests. This may have been part of the issue the whole time, but we could not clearly prove it during the earlier events because we did not yet have the Tomcat full status captures from the actual outage window. Now that we are capturing that information before reboot, the failure pattern is much more visible. That said, we also understand that you are looking for a path to stabilize the environment, not just continue collecting logs. We agree. At this point, we see a few possible improvement paths. The first option is to upgrade the vDB/vCA environment from 3.04C to 3.04F. This is important because the 3.04F upgrade path also moves the web layer from Tomcat 6 to Tomcat 9. Japan has already confirmed that Tomcat 6 can sometimes leave old long-polling threads active if disconnect notifications are not handled cleanly. Tomcat 9 is a newer web container and should provide a more modern and stable foundation for handling the type of web traffic and long-polling behavior used by vCA. This would not excuse or “fix” a runaway API client by itself, but it should make the environment less fragile than the current Tomcat 6 setup. The second option is to review whether this site should move to a line server / master server configuration. BAE-IN appears to have a relatively high number of vCA users and web/API activity for a single-server configuration. Splitting the load across a line/master configuration may help reduce the amount of vCA/web traffic handled by one Tomcat instance. The third option is operational control of API/reporting traffic. If 10.62.99.21 is confirmed to be a reporting/API system, that process should be reviewed regardless of whether we upgrade or add servers. Any API process should avoid sending too many simultaneous component-level requests, especially calls like GetPcbCompInfo, because those can be heavier requests. Throttling, batching, caching, or scheduling those jobs outside production hours may be needed. In short, the strongest current evidence points to Tomcat thread exhaustion. The immediate trigger in the latest outage captures appears to be heavy prismRest traffic from 10.62.99.21, but the longer-term stability improvement is likely either upgrading to the newer 3.04F/Tomcat 9 platform, moving toward a distributed server configuration, or both. Thank you, Jeremiah
OI
Omron internalPrivate note | May 15, 2026 at 9:58 AM
Private
Tickets with IDs 1769396 is merged into this ticket.
OI
Omron internalPrivate note | May 15, 2026 at 9:58 AM
Private
Merged from ticket 1769396 Subject: RE: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] Description: Jeremiah, The API calls from 10.62.99.21 are legit calls. They are for pulling the AOI data off the server into our databases. There is a task that runs every hour on the half hour and another that runs every 5 minutes. I gave the team that handles that a heads up that there may be changes that need made. A large number of the BAE team members are out today. Once they return next week we can start discussing a plan for how to address this situation in further detail. Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States From: AOI Tech Support - US Sent: Friday, May 15, 2026 8:44 AM To: Bowers, Samuel L (US) Cc: [email] Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] [email] [email] Warnecke, Thomas (US) ; [email] Lewis, Matthew R (US) ; Cassoday, Nolan E (US) Subject: Re: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] External Email Alert This email has been sent from an account outside of the BAE Systems network. Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros. For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar. Hi Matt, Sam, Nolan, and team, I wanted to provide a more direct update on where we are with the vCA / vDB outage investigation and what we believe the next practical options are. Based on the latest Tomcat “Complete Server Status” captures and outage logs from May 14 and May 15, we are now seeing a much clearer pattern. The server is not simply “crashing” in the traditional sense. Instead, Tomcat is reaching, or getting very close to reaching, its available request-processing thread limit. When that happens, vCA, vDB, Q-upNavi, and prismRest can appear slow, hung, or unavailable, even though the Windows services may still show as running. This became much clearer once your team started sending the full Tomcat status PDFs before rebooting the server. Earlier in the investigation, we could see the symptoms in the application logs, Oracle logs, and Windows logs, but we did not have a live view of exactly which web requests were consuming Tomcat threads at the time of failure. The full Tomcat status page gives us that view, including the active request URL, client IP, request time, and Tomcat thread state. From the latest outage captures, we saw a large number of active requests to: /prismRest/resources/GetPcbCompInfo Many of these requests appear to be coming from the same client IP address: 10.62.99.21 So one clear next step is still for BAE IT to identify what system or application is tied to that IP address. It may be a reporting tool, script, dashboard, API process, or another automated process. If that client is making many GetPcbCompInfo calls at once, those requests can consume Tomcat threads and directly contribute to the outages. That item should be corrected by stopping, throttling, rescheduling, or modifying whatever process is generating those requests. This may have been part of the issue the whole time, but we could not clearly prove it during the earlier events because we did not yet have the Tomcat full status captures from the actual outage window. Now that we are capturing that information before reboot, the failure pattern is much more visible. That said, we also understand that you are looking for a path to stabilize the environment, not just continue collecting logs. We agree. At this point, we see a few possible improvement paths. The first option is to upgrade the vDB/vCA environment from 3.04C to 3.04F. This is important because the 3.04F upgrade path also moves the web layer from Tomcat 6 to Tomcat 9. Japan has already confirmed that Tomcat 6 can sometimes leave old long-polling threads active if disconnect notifications are not handled cleanly. Tomcat 9 is a newer web container and should provide a more modern and stable foundation for handling the type of web traffic and long-polling behavior used by vCA. This would not excuse or “fix” a runaway API client by itself, but it should make the environment less fragile than the current Tomcat 6 setup. The second option is to review whether this site should move to a line server / master server configuration. BAE-IN appears to have a relatively high number of vCA users and web/API activity for a single-server configuration. Splitting the load across a line/master configuration may help reduce the amount of vCA/web traffic handled by one Tomcat instance. The third option is operational control of API/reporting traffic. If 10.62.99.21 is confirmed to be a reporting/API system, that process should be reviewed regardless of whether we upgrade or add servers. Any API process should avoid sending too many simultaneous component-level requests, especially calls like GetPcbCompInfo, because those can be heavier requests. Throttling, batching, caching, or scheduling those jobs outside production hours may be needed. In short, the strongest current evidence points to Tomcat thread exhaustion. The immediate trigger in the latest outage captures appears to be heavy prismRest traffic from 10.62.99.21, but the longer-term stability improvement is likely either upgrading to the newer 3.04F/Tomcat 9 platform, moving toward a distributed server configuration, or both. Thank you, Jeremiah On Fri, 15 May at 4:45 AM , Bowers, Samuel L (US) wrote: Good morning Jeremiah, Is this currently a high priority item for the Japan team? We understand the delay with the difference in time zones and there’s not much we can do until we hear back. I have to ask to make sure efforts are being maximized while we still have the patience of the plant. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. T: [phone] | M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: From: Lewis, Matthew R (US) Sent: Friday, May 15, 2026 4:27 AM To: Cassoday, Nolan E (US) ; AOI Tech Support - US Cc: [email] ; Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] ; [email] ; [email] ; Warnecke, Thomas (US) ; [email] ; Bowers, Samuel L (US) Subject: RE: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] And fresh event logs attached Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com From: Lewis, Matthew R (US) Sent: Friday, May 15, 2026 4:21 AM To: Cassoday, Nolan E (US) ; 'AOI Tech Support - US' Cc: [email] ; Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] ; [email] ; [email] ; Warnecke, Thomas (US) ; [email] ; Bowers, Samuel L (US) Subject: RE: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] Down again shortly after 4 am Restarted at 4:15 Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com From: Lewis, Matthew R (US) Sent: Friday, May 15, 2026 2:17 AM To: Cassoday, Nolan E (US) ; AOI Tech Support - US Cc: [email] Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] [email] [email] Warnecke, Thomas (US) ; [email] Bowers, Samuel L (US) Subject: RE: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] 10.62.99.21 is one of our bastion servers. Our personal computers are one network and the omron server and all production machines are on another. We use the bastion servers to connect from one to the other. We also remote into the Omron server from that server. Omron reported down again at 1:43 am I tried just restarting the tomcat service but the user reported that it did not help. Server restarted at about 2:10 Tomcat status, thread dump, and windows logs attached. I counted about 14 sessions in vdb before restarting Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com From: Cassoday, Nolan E (US) Sent: Thursday, May 14, 2026 9:09 PM To: AOI Tech Support - US Cc: [email] Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] [email] [email] Warnecke, Thomas (US) ; [email] Lewis, Matthew R (US) ; Bowers, Samuel L (US) Subject: RE: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] Jeremiah, Surface level look: it is not one of our review stations. I can look into more tomorrow once Sam, Matt, and I are back in plant tomorrow. Thank you Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States From: AOI Tech Support - US Sent: Thursday, May 14, 2026 9:03 PM To: Cassoday, Nolan E (US) Cc: [email] Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] [email] [email] Warnecke, Thomas (US) ; [email] Lewis, Matthew R (US) ; Bowers, Samuel L (US) Subject: Re: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] External Email Alert This email has been sent from an account outside of the BAE Systems network. Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros. For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar. Hello Nolan, Can you identify the device/server/user at IP address 10.62.99.21? During both May 14 outage captures, this IP was making many requests to: /prismRest/resources/GetPcbCompInfo These requests appear to be consuming a large number of Tomcat threads and may be the immediate cause of the outage. Please let us know? If you can stop this client, the Tomcat 200 thread pool will stop filling up. This is different then the previous Tomcat saturation outages. Let us know? On Thu, 14 May at 7:00 PM , Cassoday, Nolan E (US) wrote: Log files have been sent. Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States From: Bowers, Samuel L (US) Sent: Thursday, May 14, 2026 7:32 PM To: Cassoday, Nolan E (US) ; AOI Tech Support - US Cc: [email] Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] [email] [email] Warnecke, Thomas (US) ; [email] Lewis, Matthew R (US) Subject: RE: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] Attached is the Tomcat status page at the time of the crash. Nolan will be sending the Tomcat Dump logs when he is online next. Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: From: Cassoday, Nolan E (US) Sent: Thursday, May 14, 2026 5:36 PM To: AOI Tech Support - US ; Bowers, Samuel L (US) Cc: [email] Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] [email] [email] Warnecke, Thomas (US) ; [email] Lewis, Matthew R (US) Subject: RE: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] Jeremiah, I personally have not had the opportunity to read through your response today. I would like the Omron team to be aware thought that we had another outage today this evening. I was told at 4:50pm that we had been down for an hour at that point. We started taking actions to correct at 5:12pm. v-CA logs, SystemMonitor logs, Event Logs, TomcatThreadDump, and Tomcat status page have been sent via file transfer for review. We restarted the server about 5:20pm. Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States From: AOI Tech Support - US Sent: Thursday, May 14, 2026 10:35 AM To: Bowers, Samuel L (US) Cc: [email] Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] [email] [email] Warnecke, Thomas (US) ; [email] Cassoday, Nolan E (US) ; Lewis, Matthew R (US) Subject: Re: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] External Email Alert This email has been sent from an account outside of the BAE Systems network. Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros. For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar. Hi Matt, Sam, Nolan, and team, Japan/development has provided another update, so I wanted to summarize where things stand. At this point, there appear to be two separate items being reviewed. The first item is related to Windows Defender / Group Policy. Japan reviewed the Windows security logs we provided and believes the Defender-related policy failure is not directly related to the vCA/server slowness. Their finding is that the current GPO named NHNA.WindowsDefender.Disable is trying to disable Windows Defender, but Windows Defender tamper protection may be preventing that change from applying cleanly. Since CrowdStrike has been removed, Japan recommends that your IT team review this GPO. Specifically, they recommend changing the WinDefend service setting and the DisableAntiSpyware registry policy setting to Not Configured. The second item is the vCA / Tomcat behavior. Japan confirmed that the /vtweb/vca/IDSearchByBoard_vca request remaining active is part of the long-polling design used by the “Board ID Reading Central Monitoring” screen. However, they also confirmed that Tomcat 6 can sometimes miss disconnect notifications, which can leave old threads active. If enough of these accumulate, Tomcat can eventually stop accepting new requests once the thread pool limit is reached. Japan noted that the data they reviewed still showed available thread capacity at that time, but this is the area we are continuing to monitor. Japan also stated they have not found past cases yet where the active Tomcat configuration remained at 3.03C after an upgrade to 3.04C, and they are not recommending Tomcat timeout/keep-alive setting changes because those changes may interfere with the long-polling behavior. For now, I have not asked your team to make the GPO changes yet because I wanted to review Japan’s response first and make sure we communicate the request clearly. The recommended customer-side action would be for your IT team to review the following GPO: NHNA.WindowsDefender.Disable Japan recommends reviewing these two settings: System Services Target: WinDefend Current: Startup Type = Disabled Recommended: Change to Not Configured Registry Policy Target: Software\Policies\Microsoft\Windows Defender\DisableAntiSpyware Current: Value = 1 Recommended: Change to Not Configured Separately, I would like to continue monitoring the Tomcat status to see whether the active thread count continues to climb. The key number we are watching is the Tomcat http-8080 busy thread count. If this approaches the 200-thread limit, vCA/vDB/Q-upNavi may stop accepting new requests. We will continue monitoring the daily Tomcat “Complete Server Status” PDFs your team has been sending. Those snapshots are very helpful because they allow us to track the active Tomcat thread count, vCA session count, and the long-running IDSearchByBoard_vca connections over time. We will continue collecting that trend data and sending it to Japan/development as needed so they can review whether the long-polling buildup is stabilizing or continuing to increase. Thank you, On Thu, 14 May at 8:20 AM , Bowers, Samuel L (US) wrote: Of course! Have you heard anything else in regard to what next steps we should be taking for this issue? Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: From: AOI Tech Support - US Sent: Wednesday, May 13, 2026 9:43 AM To: Bowers, Samuel L (US) Cc: [email] Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] [email] [email] Warnecke, Thomas (US) ; [email] Cassoday, Nolan E (US) ; Lewis, Matthew R (US) Subject: Re: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] External Email Alert This email has been sent from an account outside of the BAE Systems network. Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros. For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar. Hello Sam, Thank you! I was literally about to email you and ask for this! On Wed, 13 May at 8:41 AM , Bowers, Samuel L (US) wrote: Good morning, Attached is the TomCat status page from this morning: Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: From: Lewis, Matthew R (US) Sent: Monday, May 11, 2026 9:29 AM To: AOI Tech Support - US Cc: [email] Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] [email] [email] Warnecke, Thomas (US) ; [email] Bowers, Samuel L (US) ; Cassoday, Nolan E (US) Subject: RE: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] Also, FYI, the Tomcat status page from this morning is attached. Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com From: AOI Tech Support - US Sent: Monday, May 11, 2026 9:11 AM To: Lewis, Matthew R (US) Cc: [email] Streeter, Joshua D (US) ; Miller, Todd D (US) ; [email] [email] [email] Warnecke, Thomas (US) ; [email] Bowers, Samuel L (US) ; Cassoday, Nolan E (US) Subject: Re: v-CA Webpage slow to load and hang in system when entering PCB ID [Case: 1736062] External Email Alert This email has been sent from an account outside of the BAE Systems network. Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros. For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar. Hello matt, Got the zip file, thank you. On Mon, 11 May at 8:05 AM , Lewis, Matthew R (US) wrote: Attached is the files requested. Please let me know if you get them. Nolan is out today and usually sends them. As for the second part, I got an error of “The service name is invalid” when trying the command of “net stop SoftwareUsageMetrics” My research indicated that “Software Usage metrics” is part of the User Access Logging Service. I was able to run the requested commands after stopping that service instead. Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com
33 tiny or repeated images hidden.
C
CustomerCustomer reply | May 15, 2026 at 3:40 PM
BAE Team. I think we would all agree to proceed to an escalated next step in troubleshooting this unexpected behavior. All indications point to issues with Tomcat. Tomcat is being saturated with web app sessions and not able to properly cleanout these sessions, allowing them to accumulate and reach a limit point (resulting in a slowdown or stoppage). In an effort to repair Tomcat, we recommend a specific Omron software update which includes a Tomcat version update (see below). Then we can check if a Tomcat update resolves the issue or if further steps remain needed, like rebuilding the entire server environment fresh or migrating to a multi-server environment (both of which may introduce the recommendation of the software update anyway). In this case, the software update to the entire Server / Peripheral PCs / S730 AOI systems can be performed by Omron remotely. Typically, this can be accomplished in about ~4 hours of system downtime; but this could be extended in the case of issues to be addressed. This is something Omron is likely able to perform remotely as soon as next week; but we look to BAE to advise us on system availability. Thank you. [email] [This is Brad Ward's card. Their email is [email] Their phone number is +[phone].]
Hi Brad, Appreciate you providing this information. We discussed internally and agreed that we would like to moved forward with the Omron software / Tomcat upgrade. On Monday the 25, Memorial Day - Production at our facility is not going to be running, would the Omron team be available to perform the upgrade for this Monday the 25th, Memorial Day? Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
Hi Sam. Support for this next step during the Monday holiday and next week is unavailable due to other, prearranged service project schedules. We’re currently able to reserve a remote resource during the week of Jun.1 if you’re able to commit to a date urgently. Thank you for your continued help. [This is Brad Ward's card. Their email is [email] Their phone number is +[phone].]
Brad, We would like to schedule the Omron S730 software upgrade for June 2nd at 9:00 am Eastern Day light Time. Please confirm that this time will work for your Team. Best Regards Tom Warnecke Principal Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions M: [phone]. | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States Connect with BAE Systems: [email] [email] [email] [email]
Hi Tom. This has been added to our team schedule. We’ll follow up again as the date approaches. Thank you. [This is Brad Ward's card. Their email is [email] Their phone number is +[phone].]
Good morning, I'm on standby for this morning version up to 3.04G, please let me know if you guys are ready to start the process. Best Regards [This is Saqib Toor's card. Their email is [email] Their email is [email] Their phone number is +[phone].] ________________________________
We’re still good for 9 am Eastern. I can connect to the server. If you want to take control then we need you to setup a teams meeting. We are unable to give control when we set it up. Matthew Lewis Systems Admin Sr – IT Factory Support VCP-DCV, MCSA, Security+ Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com [email]
Omron Application EngineerOmron reply | Jun 2, 2026 at 8:33 AM
Hello Matt, Could you please backup d:\program folder before we start version upgrade? I sent invitation to 3 people from teams, please add if anyone else needs to attend the meeting. Best Regards
C
CustomerCustomer reply | Jun 5, 2026 at 10:25 AM
Good morning Everyone, Thank you everyone for the efforts in regard to the server upgrade earlier this week! So far no reported issues. Attached is the Tomcat status page as of today, thought you would be interested to see what it looks like after the upgrade. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
51 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | Jun 5, 2026 at 10:38 AM
Hi Matt, Sam, Nolan, and team, Great news! The main positive finding is that the server is now running on Apache Tomcat 9.0.91, and the current Tomcat thread condition looks much healthier than what we saw before the upgrade. In this capture, Tomcat shows: Max threads: 200 Current thread count: 13 Current threads busy: 1 This is very different from the previous outage captures, where Tomcat was reaching or nearly reaching the 200-thread limit with a very high number of busy threads. In this latest status page, we do not see the same thread exhaustion condition. We also do not see a large number of active GetPcbCompInfo requests in the active request table at the time this PDF was captured. The only active request shown appears to be the local Tomcat Manager status page request. That said, we can still see that this environment has meaningful vCA/prismRest usage. The status page shows active sessions under both vtweb/vca and prismRest, and prismRest has processed a high number of requests since Tomcat startup. So the API/data collection activity is still something we should continue watching, but the important difference is that Tomcat 9 does not appear to be accumulating active worker threads in the same way in this capture. At this point, the post-upgrade status looks encouraging. We recommend continuing to monitor the Tomcat Complete Server Status page during normal operation and especially if another slowdown occurs. If the issue returns, the most important comparison will be whether Tomcat again climbs toward the 200-thread limit, or whether the behavior has changed after moving from Tomcat 6 to Tomcat 9. Regards, Jeremiah Soliz Application Engineer Direct: [phone] | [email] Omron Automation Americas 2895 Greenspoint Parkway, Suite 200, Hoffman Estates, IL 60169 automation.omron.com This is a confidential message. If received in error, delete it and notify the sender.
C
CustomerCustomer reply | Jun 5, 2026 at 6:27 PM
Good evening everyone, Unfortunately Omron went down again around 5:30 PM EST. Rebooted the server and it is operational again. I was not able to view the thread count on the Tomcat manager page as it would not load. I am kicking myself because I didn’t run the Tomcat thread dump tool before rebooting. What other logs would you like us to send? Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
57 tiny or repeated images hidden.
C
CustomerCustomer reply | Jun 8, 2026 at 9:01 AM
Good morning, No other outages occurred over the weekend. Here is the Tomcat status page as of this morning. The only sessions I am seeing are from the 10.62.99.21 server that is running the API calls. Is it normal not to see the sessions from the workstations out on the factory floor? Thank You! Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
57 tiny or repeated images hidden.
C
CustomerCustomer reply | Jun 8, 2026 at 11:30 AM
Omron went down again. Was able to get the Tomcat status page during the outage this time. Looks like a lot of session from the 10.62.99.21 server. Reaching out to the team that controls these API calls. Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
57 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | Jun 8, 2026 at 12:09 PM
Hello Sam, Please check the team responsible for these API calls to review and take corrective actions. Here are some action plan: Reducing excessive session creation and ensuring proper session reuse Validating API call frequency and eliminating unnecessary or redundant requests Reviewing session timeout settings to prevent buildup Confirming whether any changes which introduced this behavior At this point, controlling and optimizing the traffic from 10.62.99.21will be critical to preventing further outages. Saqib Toor Sr. Application Engineer Mobile: [phone] Emai: [email] Omron Automation Americas 2895 Greenspoint Parkway, Suite 200, Hoffman Estates, IL 60169 automation.omron.com [email]
C
CustomerCustomer reply | Jun 8, 2026 at 12:37 PM
Jeremiah, Saqib, As a heads up, I sent AOI logs from 6/5 and 6/8 in case they are still needed. Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
57 tiny or repeated images hidden.
C
CustomerCustomer reply | Jun 11, 2026 at 11:40 AM
Good morning, The API calls are currently under review by the team that manages them. Will provide more info as we hear back. Thank You, Sam Bowers SPV II - IT Factory Support Enterprise Shared Services IT, BAE Systems, Inc. M: [phone] | E: [email] BAE Systems, 4250 Airport Expressway, Fort Wayne, Indiana 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
63 tiny or repeated images hidden.
C
CustomerCustomer reply | Jun 12, 2026 at 9:56 AM
Good morning Omron team, As Sam noted we are working with the BAE team that manages the API calls. They looked at the code for that yesterday and they have some ideas as to why the Apache server is hitting the thread cap. I am not going to pretend like I understood all of their explanation perfectly and I am relaying the massage a day later so some of my wording may not be correct. There is a scheduled Windows task that runs every 5 minutes and another that runs once an hour. The once an hour task queries the server for anything that has not been confirmed in the last 90 days; this appears to take about 45 minutes and pulls about 6500 records. The Python code that does the work is threaded; it is supposed to use 10 threads max per how the code is being read. Possibly there is something timing out causing us to hit the thread cap or the thread workers are not freeing up in time before new ones are made causing us to hit the cap (those are the current in work ideas). Does Omron have a recommended method or optimal way for pulling the data off the AOI server with the API calls? Thank you, Nolan Cassoday Manufacturing Engineer BAE Systems Electronic Systems, Controls & Avionics Solutions T: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana 46809 United States
63 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | Jun 12, 2026 at 2:16 PM
Hi Nolan, Matt, Sam, and team, Thank you for the additional detail on the system using 10.62.99.21. Based on your description, the hourly task pulls roughly 6,500 records, takes about 45 minutes, and uses threaded Python logic with up to 10 threads. If those threads are issuing simultaneous API calls, and especially if any calls are slow, timing out, or retrying, those requests can stack up and consume Tomcat worker threads. If possible, would your team be able to share the Python script, or at least the portion that handles the API calls and threading? We would mainly want to review: how many concurrent API calls are being made whether the hourly and 5-minute jobs can overlap whether each API request has a timeout how retries are handled whether the script is repeatedly querying the same data whether the 90-day lookback can be reduced or changed to a more incremental pull If the script cannot be shared, I would recommend your team review the following items: Prevent overlapping runs If the hourly task takes 45 minutes, make sure another copy cannot start while the previous one is still running. The same should apply to the 5-minute task. Reduce concurrency for testing If the script currently allows up to 10 threads, I would suggest testing with 1 or 2 threads temporarily to see if the Omron server remains stable. Add hard request timeouts Every API call should have a timeout. If a request hangs or the server slows down, it should fail cleanly instead of waiting indefinitely. Limit retries If retries are used, they should have a limit and some backoff delay. Otherwise, a slowdown can turn into a retry storm that creates even more load. Review the 90-day query logic Pulling anything not confirmed in the last 90 days every hour may be expensive. If possible, it would be better to use a smaller incremental window, such as records changed since the last successful run. Batch and pace requests Instead of sending many component-info requests as fast as possible, add batching or short delays between groups of requests. Improve script logging It would be helpful if the script logged start/end time, number of records processed, number of API calls made, failures, retries, longest request duration, and whether another instance was already running. Hope this helps, and let me know?? Jeremiah Soliz Application Engineer Direct: [phone] | [email] Omron Automation Americas 2895 Greenspoint Parkway, Suite 200, Hoffman Estates, IL 60169 automation.omron.com This is a confidential message. If received in error, delete it and notify the sender.
OA
Omron Application EngineerOmron reply | Jun 25, 2026 at 10:53 PM
Hello team, Just curious, any updates on this? Was adjustments able to be made to that IP address to throttle API requests? It's been quiet! Let me know when you get time, thanks.
C
CustomerCustomer reply | Jun 29, 2026 at 10:58 AM
Good morning Jeremiah, The quiet has been nice.. The team spaced out the API calls so that each one finishes before the next one starts. So far, no issues have been reported since 6/12. Sam Bowers SPV II – IT Factory Support Information Technology BAE Systems, Inc. M: [phone] | E: [email] 4250 Airport Expressway, Fort Wayne, Indiana, 46809 www.baesystems.com Connect with BAE Systems: [email] [email] [email] [email] [email] [email]
attachmentattachmentattachment
24 tiny or repeated images hidden.
OA
Omron Application EngineerOmron reply | Jun 29, 2026 at 11:00 AM
Hello Sam, Thanks for the update :) I will resolve this ticket now unless we have any other concerns? Seems like everything is finally resolved?
AI
Omron Application Engineer
Ticket copilot
Connecting
Ticket#1736062 v-CA Webpage slow to load and hang in system when entering PCB ID
