Grammar Advice Requested

 My brain would appreciate some help with English grammar.  When speaking about "a portion of the swords" do verbs point to "a portion" or "swords"?  Logic tells me "a portion" while reflex points to "swords".  Advice please.