Ticket #1833227

Hello Nolan, Can you please upload the vCA logs, and the logs for the machine that you had to power off to this file transfer link? I'm curious as to what is going on as well. I personally haven't seen this, but I'll have to check with Japan and the team. Logs are a good start though. https://transfer.xmage.ninja/s/CaisDqkHHwCrZaY Let me know when you sent them? Thanks,

Hello Nolan, After reviewing the vCA logs and the AOI machine logs, this issue looks a little different than the earlier Tomcat thread-cap issue we worked through previously. At a high level, the logs appear to point more toward a VT-S730-0752-specific result/image/program state issue. In the machine logs for VT-S730-0752, we can see the system spending a long time waiting on LoadProgramXml, and the vCA logs also show failures while trying to load images/result data tied to VT-S730-0752. That lines up with the behavior you described, where the vCA webpage was still partially responding, but the inspection images, callout counts, Cancel/Complete actions, and vCA logs stopped updating correctly for that user/session. One thing we would like to confirm is whether the inspection image/result folders for the affected inspection IDs are complete or partially missing. Could you please send us a screenshot or directory listing of these two folders? G:\Images\Inspection\VT-S730-0752\[phone] G:\Images\Inspection\VT-S730-0752\[phone] We are mainly looking to confirm whether the expected image files exist in those folders, or if those folders are missing/incomplete. Also, if this happens again, it would be very helpful to run the Tomcat status/thread dump collection tool before turning off an AOI machine or restarting services. That will help us determine if Tomcat is getting tied up again, or if this is isolated to a specific machine/result/image state. At this point, we are not ruling out the API/data collection process entirely, but this event does not look as clearly like the previous API request overload issue. Since your team already reduced the API request volume significantly, that may no longer be the main contributor. However, if the API process had a backlog or was still querying older/incomplete inspection results, it could potentially add load or expose incomplete result/image data, so we would still like to keep that possibility in mind while we review the folders above. Current working theory: The vCA issue may have been caused by VT-S730-0752 having an incomplete or stuck inspection/program/result state. When vCA attempted to review that data, it could not properly load or update the expected image/result files, which caused the review screen to partially respond but not complete actions correctly. Turning off the AOI may have cleared or stopped the active machine/server interaction, allowing vCA to recover. Once we see the contents of those two image folders, we should be able to confirm whether missing or incomplete image/result files are part of the issue. Thank you, Jeremiah

Hi Sam, Yes, increasing maxThreads from 200 to 400 is reasonable to try. This setting gives Tomcat more available worker threads, so it may provide additional headroom if the server is receiving a larger number of simultaneous requests. Since you are already working with the database/API team to improve the script behavior, this could be a useful supplemental change. I would recommend backing up server.xml first, making the change, restarting Tomcat, and then monitoring the Tomcat status page afterward. If we see any negative behavior or no improvement, the change can be reverted back to 200. Jeremiah Soliz Application Engineer - Omron Inspection Systems Division [email] [phone] (ITAR authorized)