I would like to select all the ul.menu within the li's (submenu's) that don't have an "active-trail" class

HTML:

<ul class="menu">
    <li>item 1</li>
    <li class="expanded">item 2
         <ul class="menu">
             <li>sub item a</li>
             <li>sub item b</li>
             <li>sub item c</li>
         </ul>
    </li>
    <li class="expanded">item 3
         <ul class="menu">
             <li>sub item d</li>
             <li>sub item e</li>
             <li>sub item f</li>
         </ul>
    </li>
    <li class="expanded active-trail">item 4
         <ul class="menu">
             <li>sub item g</li>
             <li>sub item h</li>
             <li>sub item i</li>
         </ul>
    </li>
</ul>

JQuery:

$('#region-menu ul.menu > li.expanded:not:has(active-trail) ul.menu ').addClass('hidden')

This will do:

$('li.expanded:not(.active-trail) ul.menu').addClass('hidden');

You were close, but the selector you need inside :not() is the class .active-trail

Demo fiddle

more