[ciao-users] DAnCE: Plan Launcher throws CORBA MARSHAL exception

Mon Apr 9 10:26:09 CDT 2018

Hi,

today I enabled the ORB logging at level 10.

Here is a short snippet from the log:

2018-04-09 13:38:15.500@<local_host>@4904 at LM_DEBUG@dance_locality_manager.exe|TAO (4904|1244) - Transport[1664]::drain_queue_helper, end of data
2018-04-09 13:38:15.500@<local_host>@4904 at LM_DEBUG@dance_locality_manager.exe|TAO (4904|1244) - Transport[1664]::cleanup_queue, byte_count = 21100
2018-04-09 13:38:15.500@<local_host>@4904 at LM_DEBUG@dance_locality_manager.exe|TAO (4904|1244) - Transport[1664]::cleanup_queue, after transfer, bc = 0, all_sent = 1, ml = 0
2018-04-09 13:38:15.500@<local_host>@4904 at LM_DEBUG@dance_locality_manager.exe|TAO (4904|1244) - Transport[1664]::drain_queue_helper, byte_count = 21100, head_is_empty = 1
2018-04-09 13:38:15.500@<local_host>@4904 at LM_DEBUG@dance_locality_manager.exe|TAO (4904|1244) - Transport[1664]::drain_queue_i, helper retval = 1
2018-04-09 13:38:15.500@<local_host>@4904 at LM_DEBUG@dance_locality_manager.exe|TAO (4904|1244) - Connection_Handler[1664]::handle_input_internal, handle = 1664/1664, retval = 0
(9536|4496) [LM_DEBUG] -  13:38:15.796875 - NodeApplicationManager_impl::startLaunch - Activating NodeApplication servant
(8264|4444) [LM_TRACE] -  13:38:16.296875 - DA_NAM_ReplyHandlerImpl::startLaunch called.
(8264|4444) [LM_TRACE] -  13:38:16.296875 - DA_NAM_ReplyHandlerImpl::startLaunch - Plan ***SENSITIVE*** DomainApplication[03848570] node TestNode startLaunch succeeded, received 363 connections
(8264|4444) [LM_DEBUG] -  13:38:16.296875 - DA_NAM_ReplyHandlerImpl::Counter::on_all_completed - Plan ***SENSITIVE*** DomainApplication[03848570] successfully finished all nodes
(8264|4444) [LM_NOTICE] -  13:38:16.296875 - DomainApplicationManager_Impl::finish_startLaunch - DomainApplication[0] for plan ***SENSITIVE*** created 363 provided references.
(8264|4444) [LM_TRACE] -  13:38:16.296875 - DomainApplicationManager_Impl::finish_startLaunch - DomainApplication was successfully activated.
(7608|5104) [LM_DEBUG] -  13:38:16.859375 - Plan_Launcher_Base_Impl::start_launch - startLaunch completed...
(7608|5104) [LM_DEBUG] -  13:38:16.859375 - Plan_Launcher_Base_Impl::start_launch - DomainApplication was received from startLaunch
(7608|5104) [LM_DEBUG] -  13:38:16.859375 - Plan_Launcher_Base_Impl::launch_plan - before finishLaunch
(8264|4444) [LM_DEBUG] -  13:38:17.343750 - DomainApplication_impl::finishLaunch - Plan ***SENSITIVE*** DomainApplication[03848570] : 1 applications to finish launching
(8264|4444) [LM_TRACE] -  13:38:17.343750 - DomainApplication_Impl::finishLaunch - Plan ***SENSITIVE*** DomainApplication[03848570] node TestNode : calling finishLaunch on a node application
(8264|4444) [LM_TRACE] -  13:38:17.437500 - DomainApplication_Impl::finishLaunch - finishLaunch has been called on a node application
TAO (9536|4496) - ERROR: Could not create all profiles while extracting object [1, 4]
TAO (9536|4496) - ERROR: reference from the CDR stream.
(8264|4444) [LM_TRACE] -  13:38:17.687500 - DA_NA_ReplyHandlerImpl::finishLaunch_excep called
(8264|4444) [LM_ERROR] -  13:38:17.687500 - DA_NA_ReplyHandlerImpl::finishLaunch_excep - Plan ***SENSITIVE*** DomainApplication[03848570] node TestNode : Caught a CORBA exception: system exception, ID 'IDL:omg.org/CORBA/MARSHAL:1.0'
Unknown vendor minor code id (0), minor code = 0, completed = NO

(8264|4444) [LM_ERROR] -  13:38:17.687500 - DA_NA_ReplyHandlerImpl::Counter::on_all_completed_with_failure - Plan ***SENSITIVE*** DomainApplication[03848570] : 1 errors from node applications
(7608|5104) [LM_ERROR] -  13:38:17.687500 - Plan_Launcher_Base_Impl::finish_launch - Caught StartError exception while invoking finishLaunch: ***SENSITIVE***, 1 errors from node applications:
       TestNode - finishLaunch raised CORBA exception : system exception, ID 'IDL:omg.org/CORBA/MARSHAL:1.0'
Unknown vendor minor code id (0), minor code = 0, completed = NO

(7608|5104) [LM_ERROR] -  13:38:17.718750 - Plan_Launcher::launch_plan - Deployment failed, exception: Caught StartError exception while invoking finishLaunch: ***SENSITIVE***, 1 errors from node applications:
       TestNode - finishLaunch raised CORBA exception : system exception, ID 'IDL:omg.org/CORBA/MARSHAL:1.0'
Unknown vendor minor code id (0), minor code = 0, completed = NO

>From TAO there are two errors “Could not create all profiles while extracting object…” and “reference from the CDR stream”. Do you know what is causing these errors?

If necessary I can provide the full logfile also?

Best regards
Holger

Von: Johnny Willemsen [mailto:jwillemsen at remedy.nl] 
Gesendet: Montag, 9. April 2018 10:30
An: CIAO Users Mailing List; Haidinger, Holger
Betreff: Re: [ciao-users] DAnCE: Plan Launcher throws CORBA MARSHAL exception

Hi,
The log is not telling when exactly the marshal exception happens, probably you need to enable the ORB Logging at level 10  to get enough details to have any idea of what is happening
Regards,

Johnny WillemsenRemedy ITPostbus 81 | 6930 AB Westervoort | The Netherlandshttp://www.remedy.nl
On 04/06/2018 05:40 PM, Haidinger, Holger wrote:
Hi Will, we are using the ACE logging facility which is redirecting the log to a file. I have attached a log file as a ZIP archive. Hopefully this will work on the mailing list. The exception occurred at the end of the file. CIAO_LOG_LEVEL was not used. Best regardsHolger -----Ursprüngliche Nachricht-----Von: ciao-users [mailto:ciao-users-bounces at list.isis.vanderbilt.edu] Im Auftrag von William R. OtteGesendet: Freitag, 23. März 2018 19:15An: CIAO Users Mailing ListBetreff: Re: [ciao-users] DAnCE: Plan Launcher throws CORBA MARSHAL exception   On 23 Mar 2018, at 12:02, Haidinger, Holger wrote: Hi Will, first of all, thank you very much for your quick response! We turned up the logging by setting the following environmentvariables: set DANCE_LOG_LEVEL=10set DANCE_TRACE_ENABLE=1  Trace is a bit much, there’s a ton of overhead and you probably don’t need function call tracing at this point.  With these settings the MARSHAL exception occurred always (on faster machines).  That’s a good indication that you’re seeing a race condition of some kind, increasing the logging changes the timing of events.  Did you mean these settings or did we miss something?  Are you redirecting the log to a file?  My recollection from when I implemented that is that should be available as a feature.  That should have marginally less runtime impact over printing to stderr and then possibly redirecting to a file, as that will incur fewer context switches and will buffer less (if my recollection is correct about the implementation of the logger). Did you see any indication that the exception is coming from the node infrastructure instead of the domain infrastructure? If you’re seeing indication that it’s coming from the node level, also consider setting CIAO_LOG_LEVEL. Hth,/-Will  _______________________________________________ciao-users mailing listciao-users at list.isis.vanderbilt.eduhttp://list.isis.vanderbilt.edu/cgi-bin/mailman/listinfo/ciao-users

_______________________________________________ciao-users mailing listciao-users at list.isis.vanderbilt.eduhttp://list.isis.vanderbilt.edu/cgi-bin/mailman/listinfo/ciao-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.isis.vanderbilt.edu/pipermail/ciao-users/attachments/20180409/8e5c2937/attachment-0001.html>